{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/36374","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/36374/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/36374/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/36374/events","html_url":"https://github.com/saltstack/salt/issues/36374","id":177482503,"number":36374,"title":"Descriptor leaks in multithreaded environment","user":{"login":"szjur","id":13098524,"avatar_url":"https://avatars.githubusercontent.com/u/13098524?v=3","gravatar_id":"","url":"https://api.github.com/users/szjur","html_url":"https://github.com/szjur","followers_url":"https://api.github.com/users/szjur/followers","following_url":"https://api.github.com/users/szjur/following{/other_user}","gists_url":"https://api.github.com/users/szjur/gists{/gist_id}","starred_url":"https://api.github.com/users/szjur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szjur/subscriptions","organizations_url":"https://api.github.com/users/szjur/orgs","repos_url":"https://api.github.com/users/szjur/repos","events_url":"https://api.github.com/users/szjur/events{/privacy}","received_events_url":"https://api.github.com/users/szjur/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Cannot%20Reproduce","name":"Cannot Reproduce","color":"c7def8"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Info%20Needed","name":"Info Needed","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/37","html_url":"https://github.com/saltstack/salt/milestones/Blocked","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/37/labels","id":343741,"number":37,"title":"Blocked","description":"Issues which are blocked in some way, whether they need further discussion, clarification, justification, or are blocked for some other reason.","creator":{"login":"basepi","id":702318,"avatar_url":"https://avatars.githubusercontent.com/u/702318?v=3","gravatar_id":"","url":"https://api.github.com/users/basepi","html_url":"https://github.com/basepi","followers_url":"https://api.github.com/users/basepi/followers","following_url":"https://api.github.com/users/basepi/following{/other_user}","gists_url":"https://api.github.com/users/basepi/gists{/gist_id}","starred_url":"https://api.github.com/users/basepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basepi/subscriptions","organizations_url":"https://api.github.com/users/basepi/orgs","repos_url":"https://api.github.com/users/basepi/repos","events_url":"https://api.github.com/users/basepi/events{/privacy}","received_events_url":"https://api.github.com/users/basepi/received_events","type":"User","site_admin":false},"open_issues":579,"closed_issues":1028,"state":"open","created_at":"2013-05-28T22:24:12Z","updated_at":"2016-09-19T21:51:59Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2016-09-16T17:48:37Z","updated_at":"2016-09-19T21:54:16Z","closed_at":null,"body":"### Description of Issue/Question\r\nI ran into an issue similar to https://github.com/saltstack/salt/issues/20639. In multithreaded environment LocalClient leaks descriptor to pipes, sockets and that anon_inode:[eventfd] mentioned in #20639. But there is more to it. It seems that using cmd_async() seems to trigger the leaks. \r\n\r\nIn my use case I want to listen for events from a central thread and submit async jobs from a bunch of others. That seems to result in this massive proliferation of open handles - I think it's 4 per thread created. Those handles stay open after the thread ends.\r\n\r\nPlease see the code below.\r\n\r\n### Setup\r\nAny simple 2016.3.2 setup.\r\n\r\n### Steps to Reproduce Issue\r\nRun the code below and watch /proc/pid/fd growing.\r\n\r\n```\r\nimport os, threading, time\r\nimport salt.client\r\n\r\nclass TestThread(threading.Thread):\r\n        def run(self):\r\n                self.salt_local_client = salt.client.LocalClient()\r\n                print self.salt_local_client.cmd_async(tgt = [\"minion.name \"], fun = \"cmd.run\", arg = [\"uname -a\"], expr_form=\"list\")\r\n                time.sleep(8)\r\n                del self.salt_local_client\r\n\r\nprint \"PID: %s\" % os.getpid()\r\nfor _ in range(50):\r\n        test_thread = TestThread()\r\n        test_thread.run()\r\n        time.sleep(10)\r\n```\r\n\r\nNow change the run() function method to use cmd():\r\n\r\n```\r\nclass TestThread(threading.Thread):\r\n        def run(self):\r\n                self.salt_local_client = salt.client.LocalClient()\r\n                print self.salt_local_client.cmd(tgt = [\"minion.name \"], fun = \"cmd.run\", arg = [\"uname -a\"], expr_form=\"list\")\r\n                del self.salt_local_client\r\n```\r\n\r\nand see /proc/pid/fd stabilise at around 10-15 descriptors.\r\n\r\nI also added a debug to SaltEvent.destroy() method and in the cmd_async() variant it doesn't seem to be called. I tried calling it manually with self.salt_local_client.event.destroy() just for testing and it helps in the above example. But when that trick is applied to my use case then it somehow messes up the thread which listens for all events (without any filter).\r\n\r\n### Versions Report\r\n2016.3.2"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/248139209","html_url":"https://github.com/saltstack/salt/issues/36374#issuecomment-248139209","issue_url":"https://api.github.com/repos/saltstack/salt/issues/36374","id":248139209,"user":{"login":"szjur","id":13098524,"avatar_url":"https://avatars.githubusercontent.com/u/13098524?v=3","gravatar_id":"","url":"https://api.github.com/users/szjur","html_url":"https://github.com/szjur","followers_url":"https://api.github.com/users/szjur/followers","following_url":"https://api.github.com/users/szjur/following{/other_user}","gists_url":"https://api.github.com/users/szjur/gists{/gist_id}","starred_url":"https://api.github.com/users/szjur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szjur/subscriptions","organizations_url":"https://api.github.com/users/szjur/orgs","repos_url":"https://api.github.com/users/szjur/repos","events_url":"https://api.github.com/users/szjur/events{/privacy}","received_events_url":"https://api.github.com/users/szjur/received_events","type":"User","site_admin":false},"created_at":"2016-09-19T21:54:16Z","updated_at":"2016-09-19T21:54:16Z","body":"You are checking the right process, right? I'm not really sure how that behaves in docker. I actually forgot that I also have syndics in my environment and that affects how the client works (there is \"if self.opts['order_masters']\" etc here and there in https://github.com/saltstack/salt/blob/2016.3/salt/client/__init__.py).\r\nWhat I found out today after analysing the client code was that if I call _clean_up_subscriptions(jid) after getting a jid from cmd_async() that will stop the leaks. Looking at the code I cannot explain it why though. I believe the client calls self.event.subscribe() in the cmd_async() workflow and apparently for some reason active event subscriptions prevent garbage collector to clean up the client properly. It may depend on Python version too - unfortunately I currently need to use an ancient 2.6 version from RHEL 6.\r\nMaybe your developers can somehow make sense of that information."}}
{"actor":{"display_login":"szjur"}}