{"action":"created","issue":{"url":"https://api.github.com/repos/linkerd/linkerd/issues/1824","repository_url":"https://api.github.com/repos/linkerd/linkerd","labels_url":"https://api.github.com/repos/linkerd/linkerd/issues/1824/labels{/name}","comments_url":"https://api.github.com/repos/linkerd/linkerd/issues/1824/comments","events_url":"https://api.github.com/repos/linkerd/linkerd/issues/1824/events","html_url":"https://github.com/linkerd/linkerd/issues/1824","id":298219522,"number":1824,"title":"Memory leak - when invoking non-existing service under a heavy load","user":{"login":"MirzaMerdovic","id":16725633,"avatar_url":"https://avatars3.githubusercontent.com/u/16725633?v=4","gravatar_id":"","url":"https://api.github.com/users/MirzaMerdovic","html_url":"https://github.com/MirzaMerdovic","followers_url":"https://api.github.com/users/MirzaMerdovic/followers","following_url":"https://api.github.com/users/MirzaMerdovic/following{/other_user}","gists_url":"https://api.github.com/users/MirzaMerdovic/gists{/gist_id}","starred_url":"https://api.github.com/users/MirzaMerdovic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MirzaMerdovic/subscriptions","organizations_url":"https://api.github.com/users/MirzaMerdovic/orgs","repos_url":"https://api.github.com/users/MirzaMerdovic/repos","events_url":"https://api.github.com/users/MirzaMerdovic/events{/privacy}","received_events_url":"https://api.github.com/users/MirzaMerdovic/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/linkerd/linkerd/milestones/32","html_url":"https://github.com/linkerd/linkerd/milestone/32","labels_url":"https://api.github.com/repos/linkerd/linkerd/milestones/32/labels","id":3264269,"number":32,"title":"1.3.8","description":"","creator":{"login":"dadjeibaah","id":2197104,"avatar_url":"https://avatars2.githubusercontent.com/u/2197104?v=4","gravatar_id":"","url":"https://api.github.com/users/dadjeibaah","html_url":"https://github.com/dadjeibaah","followers_url":"https://api.github.com/users/dadjeibaah/followers","following_url":"https://api.github.com/users/dadjeibaah/following{/other_user}","gists_url":"https://api.github.com/users/dadjeibaah/gists{/gist_id}","starred_url":"https://api.github.com/users/dadjeibaah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dadjeibaah/subscriptions","organizations_url":"https://api.github.com/users/dadjeibaah/orgs","repos_url":"https://api.github.com/users/dadjeibaah/repos","events_url":"https://api.github.com/users/dadjeibaah/events{/privacy}","received_events_url":"https://api.github.com/users/dadjeibaah/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":1,"state":"open","created_at":"2018-04-12T23:21:53Z","updated_at":"2018-04-19T18:42:02Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2018-02-19T10:07:43Z","updated_at":"2018-05-22T21:13:00Z","closed_at":null,"author_association":"NONE","body":"Issue Type:\r\n\r\n- [X] Bug report\r\n- [ ] Feature request\r\n\r\n**What happened**:\r\nThe gRPC service names got changed, while load tests were running this caused linkerd to produce a memory leak.\r\n\r\n**What you expected to happen**:\r\nlinkerd memory consumption shouldn't be affected, since linkerd is aware that it received a request for a non-existing service name.\r\n\r\n`linkerd_1  | I 0216 14:36:44.672 UTC THREAD29: no available endpoints\r\nlinkerd_1  | com.twitter.finagle.NoBrokersAvailableException: No hosts are available for /svc/xxx-grpcservice, Dtab.base=[/svc=>/#/io.l5d.consul/dc1], Dtab.local=[]. Remote Info: Not Available`\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nWe were load testing our gRPC services through linkerd. Our test suite is xUnit 2.3.1 (.net core) and we ran 8-12 test scenarios in parallel. Test scenarios were basically invoking tests consecutively in specified time range (e.g. 2 hours) we no or some delay between two service calls.\r\n[Code example](https://github.com/linkerd/linkerd/files/1736134/LoadTestExample.txt) that hopefully might help.\r\nEach test scenario creates one gRPC channel.\r\n\r\n**Anything else we need to know?**:\r\nWhen we run our tests with correct service names the memory consuption is fine without increases.\r\nUploded linkerd configuration has retrying turned off, but the same thing happens with it turned on; we just wanted to see if turning it off might change something.\r\n\r\n**Environment**:\r\n- linkerd version, config files: buoyantio/linkerd:1.3.5 (buoyantio/linkerd:1.3.3)\r\n- Platform, version, and config files (Kubernetes, DC/OS, etc): docker container with Ubuntu linux\r\n[docker-compose.yaml](https://github.com/linkerd/linkerd/files/1736036/docker-compose.txt)\r\n[linkerd.yaml](https://github.com/linkerd/linkerd/files/1736039/linkerd.txt)\r\n- Cloud provider or hardware configuration: OS: Windows 10 Enterprise x64; processoer: Intel E3-1505M v5 @ 2.8Ghz (8 cpus); RAM: 32 GB\r\n"},"comment":{"url":"https://api.github.com/repos/linkerd/linkerd/issues/comments/391142535","html_url":"https://github.com/linkerd/linkerd/issues/1824#issuecomment-391142535","issue_url":"https://api.github.com/repos/linkerd/linkerd/issues/1824","id":391142535,"user":{"login":"adleong","id":3979810,"avatar_url":"https://avatars0.githubusercontent.com/u/3979810?v=4","gravatar_id":"","url":"https://api.github.com/users/adleong","html_url":"https://github.com/adleong","followers_url":"https://api.github.com/users/adleong/followers","following_url":"https://api.github.com/users/adleong/following{/other_user}","gists_url":"https://api.github.com/users/adleong/gists{/gist_id}","starred_url":"https://api.github.com/users/adleong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adleong/subscriptions","organizations_url":"https://api.github.com/users/adleong/orgs","repos_url":"https://api.github.com/users/adleong/repos","events_url":"https://api.github.com/users/adleong/events{/privacy}","received_events_url":"https://api.github.com/users/adleong/received_events","type":"User","site_admin":false},"created_at":"2018-05-22T21:12:59Z","updated_at":"2018-05-22T21:12:59Z","author_association":"MEMBER","body":"@MirzaMerdovic That's correct.  Linkerd should maintain one connection to Consul per service that is watching (regardless of if that service exists or not).  It should be independent of the number of requests.  If you are seeing new connections for each request to the same service, that is a bug."}}
{"actor":{"display_login":"adleong"}}