{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/23240","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/23240/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/23240/comments","events_url":"https://api.github.com/repos/docker/docker/issues/23240/events","html_url":"https://github.com/docker/docker/pull/23240","id":158391748,"number":23240,"title":"Use @cpuguy83's patch and add a global cleanup lock","user":{"login":"etamponi","id":578612,"avatar_url":"https://avatars.githubusercontent.com/u/578612?v=3","gravatar_id":"","url":"https://api.github.com/users/etamponi","html_url":"https://github.com/etamponi","followers_url":"https://api.github.com/users/etamponi/followers","following_url":"https://api.github.com/users/etamponi/following{/other_user}","gists_url":"https://api.github.com/users/etamponi/gists{/gist_id}","starred_url":"https://api.github.com/users/etamponi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/etamponi/subscriptions","organizations_url":"https://api.github.com/users/etamponi/orgs","repos_url":"https://api.github.com/users/etamponi/repos","events_url":"https://api.github.com/users/etamponi/events{/privacy}","received_events_url":"https://api.github.com/users/etamponi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/group/windows","name":"group/windows","color":"F6FBFF"},{"url":"https://api.github.com/repos/docker/docker/labels/status/1-design-review","name":"status/1-design-review","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-06-03T15:21:28Z","updated_at":"2016-09-04T16:18:05Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/docker/docker/pulls/23240","html_url":"https://github.com/docker/docker/pull/23240","diff_url":"https://github.com/docker/docker/pull/23240.diff","patch_url":"https://github.com/docker/docker/pull/23240.patch"},"body":"I got stuck in #5618 and #17443 for a week without finding a good solution, except\r\nfor the great @cpuguy83 's PR #23178 that unlocked at least the readers (ps, inspect)\r\nbut could not avoid NMI watchdog soft lockups nor unregister_netdevice errors.\r\n\r\nI found the issue #19758 and https://github.com/docker/docker/issues/19758#issuecomment-182305310\r\n gave me the idea: since it only occours for concurrent cleanups, why not\r\nputting a Global Cleanup Lock in place? (I come from a Python background, does it show? 😄  )\r\n\r\nIt works well, I have used a modified version of the docker stress tool that you can find\r\nat https://github.com/etamponi/docker-stress . I used this configuration:\r\n\r\n    docker-stress --containers 10000 --concurrent 200 --kill 3s busybox\r\n\r\nwith the `stress.json` file that I put in that repo. I tried several times, and no issues of\r\nsort happened. Using the same command on v1.11.2, or v1.10 and any kernel from 3.16 to 4.5 causes\r\nNMI watchdog lockups and/or unregister_netdevice errors.\r\n\r\nOf course this is a POC patch that shows where the issue is. From here we can try to add finer\r\nlocks, only where they are needed (perhaps just around the `releaseNetwork` call). But until the\r\nkernel data race is fixed I fear we cannot get anything less than a \"global\" lock somewhere.\r\n\r\nBeautiful animal: an hummigbird :)\r\n\r\n![screen shot 2016-06-03 at 17 20 08](https://cloud.githubusercontent.com/assets/578612/15783690/73713f7a-29af-11e6-9c8b-ff519b668c6a.png)\r\n\r\nSigned-off-by: Emanuele Tamponi <emanuele.tamponi@gmail.com>"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/244612431","html_url":"https://github.com/docker/docker/pull/23240#issuecomment-244612431","issue_url":"https://api.github.com/repos/docker/docker/issues/23240","id":244612431,"user":{"login":"vdemeester","id":6508,"avatar_url":"https://avatars.githubusercontent.com/u/6508?v=3","gravatar_id":"","url":"https://api.github.com/users/vdemeester","html_url":"https://github.com/vdemeester","followers_url":"https://api.github.com/users/vdemeester/followers","following_url":"https://api.github.com/users/vdemeester/following{/other_user}","gists_url":"https://api.github.com/users/vdemeester/gists{/gist_id}","starred_url":"https://api.github.com/users/vdemeester/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdemeester/subscriptions","organizations_url":"https://api.github.com/users/vdemeester/orgs","repos_url":"https://api.github.com/users/vdemeester/repos","events_url":"https://api.github.com/users/vdemeester/events{/privacy}","received_events_url":"https://api.github.com/users/vdemeester/received_events","type":"User","site_admin":false},"created_at":"2016-09-04T16:18:05Z","updated_at":"2016-09-04T16:18:05Z","body":"@cpuguy83 what is the status here ? 👼"}}
{"actor":{"display_login":"vdemeester"}}