{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/267","repository_url":"https://api.github.com/repos/mesosphere/marathon-lb","labels_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/267/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/267/comments","events_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/267/events","html_url":"https://github.com/mesosphere/marathon-lb/issues/267","id":166167326,"number":267,"title":"Race condition causing not released lock and too many HAProxy processes","user":{"login":"mlowicki","id":97633,"avatar_url":"https://avatars.githubusercontent.com/u/97633?v=3","gravatar_id":"","url":"https://api.github.com/users/mlowicki","html_url":"https://github.com/mlowicki","followers_url":"https://api.github.com/users/mlowicki/followers","following_url":"https://api.github.com/users/mlowicki/following{/other_user}","gists_url":"https://api.github.com/users/mlowicki/gists{/gist_id}","starred_url":"https://api.github.com/users/mlowicki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlowicki/subscriptions","organizations_url":"https://api.github.com/users/mlowicki/orgs","repos_url":"https://api.github.com/users/mlowicki/repos","events_url":"https://api.github.com/users/mlowicki/events{/privacy}","received_events_url":"https://api.github.com/users/mlowicki/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-07-18T19:00:11Z","updated_at":"2016-07-18T22:30:56Z","closed_at":null,"body":"Hi,\r\n\r\nInitially I've asked on Marathon [mailing list](https://groups.google.com/forum/?hl=en#!topic/marathon-framework/6wInFoQGu7w) but gather more data and since I've discovered that https://github.com/mesosphere/marathon-lb/blob/master/service/haproxy/run isn't provided by HAProxy but Marathon-lb then issue probably is caused by later.\r\n\r\nLooks like lock isn't released and we've some hanging calls to flock:\r\n```\r\nroot@lb2i:/marathon-lb/service/haproxy# ps aux | grep [f]lock\r\nroot       385  0.0  0.0   8464   868 ?        S    Jul13   0:00 flock 200\r\nroot       393  0.0  0.0   8464   856 ?        S    Jul13   0:00 flock 200\r\nroot       790  0.0  0.0   8464   856 ?        S    Jul13   0:00 flock 200\r\nroot      4951  0.0  0.0   8464   936 ?        S    Jul13   0:00 flock 200\r\nroot      6551  0.0  0.0   8464   856 ?        S    Jul13   0:00 flock 200\r\nroot     11841  0.0  0.0   8464   868 ?        S    Jul13   0:00 flock 200\r\nroot     11857  0.0  0.0   8464   864 ?        S    Jul13   0:00 flock 200\r\nroot     17515  0.0  0.0   8464   832 ?        S    Jul13   0:00 flock 200\r\nroot     17543  0.0  0.0   8464   796 ?        S    Jul13   0:00 flock 200\r\nroot     23167  0.0  0.0   8464   876 ?        S    Jul13   0:00 flock 200\r\nroot     24277  0.0  0.0   8464   856 ?        S    Jul13   0:00 flock 200\r\nroot     27799  0.0  0.0   8464   872 ?        S    Jul13   0:00 flock 200\r\nroot     27993  0.0  0.0   8464   880 ?        S    Jul13   0:00 flock 200\r\nroot     30333  0.0  0.0   8464   936 ?        S    Jul13   0:00 flock 200\r\nroot     34368  0.0  0.0   8464   884 ?        S    Jul13   0:00 flock 200\r\nroot     47233  0.0  0.0   8464   880 ?        S    Jul13   0:00 flock 200\r\nroot     48034  0.0  0.0   8464   936 ?        S    Jul13   0:00 flock 200\r\nroot     49056  0.0  0.0   8464   828 ?        S    Jul13   0:00 flock 200\r\n```\r\n\r\n```\r\nroot@lb2i:/marathon-lb/service/haproxy# flock --wait 5 ./run \"echo d\"\r\nroot@lb2i:/marathon-lb/service/haproxy# echo $?\r\n1\r\n```\r\n\r\n```\r\n> lsof ./run\r\nCOMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME\r\nflock     385 root  200r   REG  253,4     1228 8426486 ./run\r\nflock     393 root  200r   REG  253,4     1228 8426486 ./run\r\nflock     790 root  200r   REG  253,4     1228 8426486 ./run\r\nflock    4951 root  200r   REG  253,4     1228 8426486 ./run\r\nflock    6551 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   11841 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   11857 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   17515 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   17543 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   23167 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   24277 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   27799 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27800 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27801 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27802 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27803 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27804 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27805 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27806 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27807 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27808 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27809 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27810 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27811 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27812 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27813 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27814 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27815 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27816 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27817 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27818 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27819 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27820 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27821 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27822 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27823 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27824 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27825 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27826 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27827 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27828 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27829 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27830 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27831 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27832 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27833 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27834 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27835 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27836 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27837 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27838 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27839 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27840 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27841 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27842 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27843 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27844 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27845 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27846 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27847 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27914 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27915 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27916 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27917 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27918 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27919 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27920 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27921 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27922 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27923 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27924 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27925 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27926 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27927 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27928 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27929 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27930 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27931 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27932 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27933 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27934 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27935 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27936 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27937 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27938 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27939 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27940 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27941 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27942 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27943 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27944 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27945 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27946 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27947 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27948 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27949 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27950 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27951 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27952 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27953 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27954 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27955 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27956 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27957 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27958 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27959 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27960 root  200r   REG  253,4     1228 8426486 ./run\r\nhaproxy 27961 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   27993 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   30333 root  200r   REG  253,4     1228 8426486 ./run\r\nrun     34362 root  200r   REG  253,4     1228 8426486 ./run\r\nrun     34362 root  255r   REG  253,4     1228 8426486 ./run\r\nflock   34368 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   47233 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   48034 root  200r   REG  253,4     1228 8426486 ./run\r\nflock   49056 root  200r   REG  253,4     1228 8426486 ./run\r\n```\r\n\r\nI'm running:\r\n```\r\ncat /proc/version\r\nLinux version 4.5.0-0.bpo.1-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Debian 4.5.1-1~bpo8+1 (2016-04-20) \r\n```\r\n\r\nInitially we've suspected it's related to `nbproc` we're using from HAProxy but even without it we've reached the state when we had more than one HAProxy process (right after changing configuration, killing old containers and then Marathon starts new one). It doesn't reproduce always though so probably there is some kind of race condition\r\n\r\nLists of HAProxy processes (there are 96 but should be 48 but we had also box with almost 500 HAProxy processes) - https://gist.github.com/mlowicki/1b3672db46c2dcd26f1db7bba004ad99."},"comment":{"url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/comments/233478849","html_url":"https://github.com/mesosphere/marathon-lb/issues/267#issuecomment-233478849","issue_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/267","id":233478849,"user":{"login":"mlowicki","id":97633,"avatar_url":"https://avatars.githubusercontent.com/u/97633?v=3","gravatar_id":"","url":"https://api.github.com/users/mlowicki","html_url":"https://github.com/mlowicki","followers_url":"https://api.github.com/users/mlowicki/followers","following_url":"https://api.github.com/users/mlowicki/following{/other_user}","gists_url":"https://api.github.com/users/mlowicki/gists{/gist_id}","starred_url":"https://api.github.com/users/mlowicki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlowicki/subscriptions","organizations_url":"https://api.github.com/users/mlowicki/orgs","repos_url":"https://api.github.com/users/mlowicki/repos","events_url":"https://api.github.com/users/mlowicki/events{/privacy}","received_events_url":"https://api.github.com/users/mlowicki/received_events","type":"User","site_admin":false},"created_at":"2016-07-18T22:30:56Z","updated_at":"2016-07-18T22:30:56Z","body":"Yeah, noticed it and seems as you mentioned that could be improved a bit (theoretically still can be vulnerable to my scenario - there are also 2 calls to `sleep` in https://github.com/mesosphere/marathon-lb/blob/master/service/haproxy/run#L41 but as it happens randomly it might be somehow bypassed).\r\n\r\nNot sure if we should check only that new processes have been added but maybe also that old ones are gone to avoid my scenario? I might be that `$(cat $PIDFILE)` doesn't report processes waiting to be drained off but still present in OS' process table?"}}
{"actor":{"display_login":"mlowicki"}}