{"action":"opened","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/33585","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/33585/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/33585/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/33585/events","html_url":"https://github.com/saltstack/salt/issues/33585","id":157221106,"number":33585,"title":"inode usage 100%","user":{"login":"jakehilton","id":243169,"avatar_url":"https://avatars.githubusercontent.com/u/243169?v=3","gravatar_id":"","url":"https://api.github.com/users/jakehilton","html_url":"https://github.com/jakehilton","followers_url":"https://api.github.com/users/jakehilton/followers","following_url":"https://api.github.com/users/jakehilton/following{/other_user}","gists_url":"https://api.github.com/users/jakehilton/gists{/gist_id}","starred_url":"https://api.github.com/users/jakehilton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakehilton/subscriptions","organizations_url":"https://api.github.com/users/jakehilton/orgs","repos_url":"https://api.github.com/users/jakehilton/repos","events_url":"https://api.github.com/users/jakehilton/events{/privacy}","received_events_url":"https://api.github.com/users/jakehilton/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-27T14:38:08Z","updated_at":"2016-05-27T14:38:08Z","closed_at":null,"body":"### Description of Issue/Question\r\nRunning 2016.3.0 proper for a few days and two of my 10 masters show inode usage of 96% and 100%. Looking at the culprit I found that `/var/cache/salt/master/jobs` is the offending dir. \r\n\r\nRunning `salt-run jobs.list_jobs` or `salt-run jobs.active` hangs or sits that way for 5+ minutes. \r\n\r\nNeedless to say my systems where this is happening is rendering them less than useful. \r\n\r\n### Setup\r\n```\r\n# df -i\r\nFilesystem      Inodes   IUsed  IFree IUse% Mounted on\r\nudev            956144     387 955757    1% /dev\r\ntmpfs           957439     314 957125    1% /run\r\n/dev/xvda1     3276800 3274437   2363  100% /\r\nnone            957439       2 957437    1% /sys/fs/cgroup\r\nnone            957439       1 957438    1% /run/lock\r\nnone            957439       8 957431    1% /run/shm\r\nnone            957439       2 957437    1% /run/user\r\n```\r\n```\r\n# df -h\r\nFilesystem      Size  Used Avail Use% Mounted on\r\nudev            3.7G   12K  3.7G   1% /dev\r\ntmpfs           748M  360K  748M   1% /run\r\n/dev/xvda1       50G   18G   30G  37% /\r\nnone            4.0K     0  4.0K   0% /sys/fs/cgroup\r\nnone            5.0M     0  5.0M   0% /run/lock\r\nnone            3.7G   28K  3.7G   1% /run/shm\r\nnone            100M     0  100M   0% /run/user\r\n```\r\nMaster sls\r\n```\r\nhash_type: sha256\r\nkeep_jobs: 1\r\nopen_mode: True\r\nstate_output: mixed\r\nstate_verbose: False\r\ntimeout: 30\r\ntransport: tcp\r\nworker_threads: 10\r\n\r\nfile_roots:\r\n  base:\r\n    - /etc/salt/git/salt\r\n    - /etc/salt/git/salt/states\r\n\r\npillar_roots:\r\n  base:\r\n    - /etc/salt/git/salt/pillar\r\n```\r\n\r\n### Steps to Reproduce Issue\r\nRunning between 300-600 minions against the problematic masters.\r\n\r\n### Versions Report\r\n```\r\n# salt --versions-report\r\nSalt Version:\r\n           Salt: 2016.3.0\r\n\r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 2.2\r\n          gitdb: 0.5.4\r\n      gitpython: 0.3.2 RC1\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: 0.22\r\n           Mako: 0.9.1\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.7\r\n   mysql-python: 1.2.3\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.10\r\n          PyZMQ: 14.0.1\r\n           RAET: Not Installed\r\n          smmap: 0.8.2\r\n        timelib: Not Installed\r\n        Tornado: 4.3\r\n            ZMQ: 4.0.4\r\n\r\nSystem Versions:\r\n           dist: Ubuntu 14.04 trusty\r\n        machine: x86_64\r\n        release: 3.13.0-85-generic\r\n         system: Linux\r\n        version: Ubuntu 14.04 trusty\r\n```\r\n"}}
{"actor":{"display_login":"jakehilton"}}