{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/38386","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/38386/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/38386/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/38386/events","html_url":"https://github.com/saltstack/salt/issues/38386","id":196816031,"number":38386,"title":"cmd.run doesn't kill completely when there are multiple processes spawned - causes cmd.run to hang after command exit","user":{"login":"onlyanegg","id":7731128,"avatar_url":"https://avatars.githubusercontent.com/u/7731128?v=3","gravatar_id":"","url":"https://api.github.com/users/onlyanegg","html_url":"https://github.com/onlyanegg","followers_url":"https://api.github.com/users/onlyanegg/followers","following_url":"https://api.github.com/users/onlyanegg/following{/other_user}","gists_url":"https://api.github.com/users/onlyanegg/gists{/gist_id}","starred_url":"https://api.github.com/users/onlyanegg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/onlyanegg/subscriptions","organizations_url":"https://api.github.com/users/onlyanegg/orgs","repos_url":"https://api.github.com/users/onlyanegg/repos","events_url":"https://api.github.com/users/onlyanegg/events{/privacy}","received_events_url":"https://api.github.com/users/onlyanegg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-21T00:26:25Z","updated_at":"2016-12-21T21:53:01Z","closed_at":null,"body":"### Description of Issue/Question\r\nI have a bash script that tails a file until a string is found. I use it to make sure that services are fully up.\r\nI call this via cmd.run with a timeout. This is what the process table looks like when it’s running: \r\n```\r\n$ salt-call state.apply common.tail_until_test &\r\n[1] 12297\r\n$ ps -ef | grep tai[l]\r\nroot     12297  7939  2 15:50 pts/0    00:00:00 /usr/bin/python /bin/salt-call state.apply common.tail_until_test\r\nroot     12544 12297  0 15:50 pts/0    00:00:00 /bin/bash /usr/local/bin/tail_until /tmp/test_tail_until.log test message\r\nroot     12546 12544  0 15:50 pts/0    00:00:00 tail -f -n 0 --pid 12544 /tmp/test_tail_until.log\r\n```\r\nAnd this is what it looks like after it times out:\r\n```\r\n$ ps -ef | grep tai[l]\r\nroot     12297  7939  2 15:50 pts/0    00:00:00 /usr/bin/python /bin/salt-call state.apply common.tail_until_test\r\nroot     12544 12297  0 15:50 pts/0    00:00:00 [tail_until] <defunct>\r\nroot     12546     1  0 15:50 pts/0    00:00:00 tail -f -n 0 --pid 12544 /tmp/test_tail_until.log\r\n```\r\nThis causes salt-call to hang well after the timeout.\r\n\r\nHowever, when I call this script on the terminal, and send a SIGTERM (this is what cmd.run is documented to do after a timeout), the process table is clean.\r\n```\r\n$ /usr/local/bin/tail_until /tmp/test.log test message &\r\n[1] 10385\r\n$ ps -ef | grep tai[l]\r\nroot     10385  8096  0 05:54 pts/0    00:00:00 /bin/bash /usr/local/bin/tail_until /tmp/test.log test message\r\nroot     10386 10385  0 05:54 pts/0    00:00:00 tail -f -n 0 --pid 10385 /tmp/test.log\r\n$ kill -TERM 10385\r\n$ ps -ef | grep tai[l]\r\n[1]+  Terminated              /usr/local/bin/tail_until /tmp/test.log test message\r\n```\r\nMy best guess as to what is happening is that cmd.run is not properly killing the original process. Since tail’s parent is still in the process table, tail doesn’t exit.\r\n\r\nThis seems like this could be by design. Salt (or what ever is actually doing the killing) wants to make sure that all children are cleaned up before exiting.\r\n\r\n### Setup\r\n`tail_until_test.sls`\r\n```sls\r\ntest_tail_until:\r\n  cmd.run:\r\n    - name: /usr/local/bin/tail_until /tmp/test_tail_until.log 'test message'\r\n    - timeout: 5\r\n```\r\n`/usr/local/bin/tail_until`\r\n```sh\r\n#!/bin/bash\r\n#\r\n\r\nshow_help() {\r\n  cat << EOF\r\nTail file until pattern is found.\r\n\r\nUsage:\r\n`basename $0` '<file>' '<pattern>'\r\nEOF\r\n}\r\n\r\nmain() {\r\n  pipe=/tmp/tail_until.pipe\r\n  [[ -p $pipe ]] || mkfifo $pipe\r\n  tail -f -n 0 --pid $$ $file & >> $pipe\r\n  while read line; do\r\n    [[ \"$line\" == *\"$pattern\"* ]] && exit 0\r\n  done < $pipe\r\n}\r\n\r\ncase $1 in\r\n  -h|--help)\r\n    show_help\r\n    exit\r\n    ;;\r\n  *)\r\n    file=$1 && shift\r\n    pattern=$@\r\n    ;;\r\nesac\r\n\r\nif [ $0 != \"-bash\" ]; then\r\n  main\r\nfi\r\n```\r\n\r\n### Versions Report\r\n```\r\n$ salt-call --versions-report\r\nSalt Version:\r\n           Salt: 2016.11.1\r\n\r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 2.6.0\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.8\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.5 (default, Sep 15 2016, 22:37:39)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 15.3.0\r\n           RAET: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.1.4\r\n\r\nSystem Versions:\r\n           dist: centos 7.2.1511 Core\r\n        machine: x86_64\r\n        release: 3.10.0-327.36.3.el7.x86_64\r\n         system: Linux\r\n        version: CentOS Linux 7.2.1511 Core\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/268650798","html_url":"https://github.com/saltstack/salt/issues/38386#issuecomment-268650798","issue_url":"https://api.github.com/repos/saltstack/salt/issues/38386","id":268650798,"user":{"login":"Ch3LL","id":6886822,"avatar_url":"https://avatars.githubusercontent.com/u/6886822?v=3","gravatar_id":"","url":"https://api.github.com/users/Ch3LL","html_url":"https://github.com/Ch3LL","followers_url":"https://api.github.com/users/Ch3LL/followers","following_url":"https://api.github.com/users/Ch3LL/following{/other_user}","gists_url":"https://api.github.com/users/Ch3LL/gists{/gist_id}","starred_url":"https://api.github.com/users/Ch3LL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ch3LL/subscriptions","organizations_url":"https://api.github.com/users/Ch3LL/orgs","repos_url":"https://api.github.com/users/Ch3LL/repos","events_url":"https://api.github.com/users/Ch3LL/events{/privacy}","received_events_url":"https://api.github.com/users/Ch3LL/received_events","type":"User","site_admin":false},"created_at":"2016-12-21T21:53:01Z","updated_at":"2016-12-21T21:53:01Z","body":"@onlyanegg i'm having a hard time replicating this using your exact same state and script.\r\n\r\nThis is what it looks like during execution:\r\n\r\n```\r\n[root@localhost salt]# ps -ef | grep tai[l]\r\nroot      9416  3005 11 06:51 pts/0    00:00:00 /usr/bin/python2 /bin/salt-call state.apply common.tail_until_test\r\nroot      9489  9416  0 06:51 pts/0    00:00:00 /bin/bash /usr/local/bin/tail_until /tmp/test_tail_until.log test message\r\n```\r\n\r\nAnd after:\r\n\r\n```\r\n[root@localhost salt]# ps -ef | grep tai[l]\r\n[root@localhost salt]# \r\n```\r\n\r\nAre you seeing this on all your minions?\r\nDid you recently just start seeing this and if so what changed in your environment?\r\nany other information to help me replicate?"}}
{"actor":{"display_login":"Ch3LL"}}