{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/serving/issues/1215","repository_url":"https://api.github.com/repos/tensorflow/serving","labels_url":"https://api.github.com/repos/tensorflow/serving/issues/1215/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/serving/issues/1215/comments","events_url":"https://api.github.com/repos/tensorflow/serving/issues/1215/events","html_url":"https://github.com/tensorflow/serving/issues/1215","id":391360543,"node_id":"MDU6SXNzdWUzOTEzNjA1NDM=","number":1215,"title":"Sharp increase in memory usage -> server is killed","user":{"login":"echan00","id":6287299,"node_id":"MDQ6VXNlcjYyODcyOTk=","avatar_url":"https://avatars0.githubusercontent.com/u/6287299?v=4","gravatar_id":"","url":"https://api.github.com/users/echan00","html_url":"https://github.com/echan00","followers_url":"https://api.github.com/users/echan00/followers","following_url":"https://api.github.com/users/echan00/following{/other_user}","gists_url":"https://api.github.com/users/echan00/gists{/gist_id}","starred_url":"https://api.github.com/users/echan00/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/echan00/subscriptions","organizations_url":"https://api.github.com/users/echan00/orgs","repos_url":"https://api.github.com/users/echan00/repos","events_url":"https://api.github.com/users/echan00/events{/privacy}","received_events_url":"https://api.github.com/users/echan00/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-12-15T09:18:30Z","updated_at":"2018-12-19T07:05:59Z","closed_at":"2018-12-18T19:57:29Z","author_association":"NONE","body":"Every so often I have found my tensorflow server killing itself due to memory usage hitting capacity. I can't quite figure out how or why this happening because it only happens occasionally and my process to use the server to infer files sequentially is identical between multiple servers and runtimes.\r\n\r\nMy guess is the issue is related to the files itself (the only non-constant), has anybody run into similar problems?  For the sake of context, I am using my tf server to translate text in these files."},"comment":{"url":"https://api.github.com/repos/tensorflow/serving/issues/comments/448491324","html_url":"https://github.com/tensorflow/serving/issues/1215#issuecomment-448491324","issue_url":"https://api.github.com/repos/tensorflow/serving/issues/1215","id":448491324,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0ODQ5MTMyNA==","user":{"login":"masroorhasan","id":802678,"node_id":"MDQ6VXNlcjgwMjY3OA==","avatar_url":"https://avatars1.githubusercontent.com/u/802678?v=4","gravatar_id":"","url":"https://api.github.com/users/masroorhasan","html_url":"https://github.com/masroorhasan","followers_url":"https://api.github.com/users/masroorhasan/followers","following_url":"https://api.github.com/users/masroorhasan/following{/other_user}","gists_url":"https://api.github.com/users/masroorhasan/gists{/gist_id}","starred_url":"https://api.github.com/users/masroorhasan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/masroorhasan/subscriptions","organizations_url":"https://api.github.com/users/masroorhasan/orgs","repos_url":"https://api.github.com/users/masroorhasan/repos","events_url":"https://api.github.com/users/masroorhasan/events{/privacy}","received_events_url":"https://api.github.com/users/masroorhasan/received_events","type":"User","site_admin":false},"created_at":"2018-12-19T07:05:59Z","updated_at":"2018-12-19T07:05:59Z","author_association":"NONE","body":"Hi @gautamvasudevan, @echan00 \r\n\r\nI've also noticed a memory leak problem on server side thats loading an inception model (similar to mnist).\r\n\r\nThe memory leak problem on inception model can be traced back to synchronous vs asynchronous gRPC requests to the model server (i.e. parallel vs sequential prediction requests to inception graph hosted on tensorflow serving).\r\n\r\nAsync requests (using `stub.Predict.future(request, 30.0)`) to model server:\r\n![image](https://user-images.githubusercontent.com/802678/50021389-c5467200-ff8d-11e8-9eed-7037d73cfb96.png)\r\n\r\n\r\nSync request (`stub.Predict(request, 30.0)`) to model server:\r\n![image](https://user-images.githubusercontent.com/802678/50021418-e14a1380-ff8d-11e8-93c2-d5e8df54403d.png)\r\n\r\nSync requests distributed parallelized using threadpool (of size 8):\r\n![image](https://user-images.githubusercontent.com/802678/50155672-e6dd8b80-02f6-11e9-8997-eec9eb050a82.png)\r\n\r\nThis issue has been reported sparsely in github issues and some SO posts, but this is a big concern as a common inception client example with async will lead to server memory leak. IMO this issue should be re-opened for further discussion.\r\n\r\n@gautamvasudevan let me know if you need information, i've also reproduced this issue using `faster_rcnn_inception_resnet_v2_atrous_oid` from model zoo (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md). I can provide sample scripts and more examples. "}}
{"actor":{"display_login":"masroorhasan"}}