{"action":"created","issue":{"url":"https://api.github.com/repos/docker/swarm/issues/2178","repository_url":"https://api.github.com/repos/docker/swarm","labels_url":"https://api.github.com/repos/docker/swarm/issues/2178/labels{/name}","comments_url":"https://api.github.com/repos/docker/swarm/issues/2178/comments","events_url":"https://api.github.com/repos/docker/swarm/issues/2178/events","html_url":"https://github.com/docker/swarm/issues/2178","id":151362711,"number":2178,"title":"runtime: failed to create new OS thread (have 40325 already","user":{"login":"shenshouer","id":1491181,"avatar_url":"https://avatars.githubusercontent.com/u/1491181?v=3","gravatar_id":"","url":"https://api.github.com/users/shenshouer","html_url":"https://github.com/shenshouer","followers_url":"https://api.github.com/users/shenshouer/followers","following_url":"https://api.github.com/users/shenshouer/following{/other_user}","gists_url":"https://api.github.com/users/shenshouer/gists{/gist_id}","starred_url":"https://api.github.com/users/shenshouer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shenshouer/subscriptions","organizations_url":"https://api.github.com/users/shenshouer/orgs","repos_url":"https://api.github.com/users/shenshouer/repos","events_url":"https://api.github.com/users/shenshouer/events{/privacy}","received_events_url":"https://api.github.com/users/shenshouer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/swarm/labels/kind/bug","name":"kind/bug","color":"207de5"},{"url":"https://api.github.com/repos/docker/swarm/labels/priority/P1","name":"priority/P1","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/docker/swarm/milestones/14","html_url":"https://github.com/docker/swarm/milestones/1.2.3","labels_url":"https://api.github.com/repos/docker/swarm/milestones/14/labels","id":1751960,"number":14,"title":"1.2.3","description":"","creator":{"login":"nishanttotla","id":1872537,"avatar_url":"https://avatars.githubusercontent.com/u/1872537?v=3","gravatar_id":"","url":"https://api.github.com/users/nishanttotla","html_url":"https://github.com/nishanttotla","followers_url":"https://api.github.com/users/nishanttotla/followers","following_url":"https://api.github.com/users/nishanttotla/following{/other_user}","gists_url":"https://api.github.com/users/nishanttotla/gists{/gist_id}","starred_url":"https://api.github.com/users/nishanttotla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nishanttotla/subscriptions","organizations_url":"https://api.github.com/users/nishanttotla/orgs","repos_url":"https://api.github.com/users/nishanttotla/repos","events_url":"https://api.github.com/users/nishanttotla/events{/privacy}","received_events_url":"https://api.github.com/users/nishanttotla/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":0,"state":"open","created_at":"2016-05-06T16:27:49Z","updated_at":"2016-05-06T16:49:09Z","due_on":"2016-05-20T07:00:00Z","closed_at":null},"comments":2,"created_at":"2016-04-27T11:27:56Z","updated_at":"2016-05-06T17:21:12Z","closed_at":null,"body":"swarm manager crash . the log:\r\n\r\n```\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: runtime: failed to create new OS thread (have 40325 already; errno=11)\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: fatal error: newosproc\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: runtime stack:\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: runtime.throw(0xfeaa30, 0x9)\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: /usr/local/go/src/runtime/panic.go:530 +0x90\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: runtime.newosproc(0xc9010c2800, 0xc8ffff1fc0)\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: /usr/local/go/src/runtime/os1_linux.go:149 +0x18c\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: runtime.newm(0x11c73d0, 0xc820043500)\r\nApr 27 17:51:43 worker-x-x-x-121 swarm[11729]: /usr/local/go/src/runtime/proc.go:1513 +0x135\r\n```\r\n\r\nswarm info:\r\n```\r\nworker-x-x-x-121 system # docker -H :4000 info\r\nContainers: 1524\r\n Running: 1512\r\n Paused: 0\r\n Stopped: 12\r\nImages: 541\r\nServer Version: swarm/1.2.0\r\nRole: replica\r\nPrimary: x.x.x.122:4000\r\nStrategy: spread\r\nFilters: health, port, dependency, affinity, constraint\r\nNodes: 12\r\n worker-x-x-x-111: x.x.x.111:2375\r\n  └ Status: Healthy\r\n  └ Containers: 142\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-112: x.x.x.112:2375\r\n  └ Status: Healthy\r\n  └ Containers: 141\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-113: x.x.x.113:2375\r\n  └ Status: Healthy\r\n  └ Containers: 141\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-115: x.x.x.115:2375\r\n  └ Status: Healthy\r\n  └ Containers: 140\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-116: x.x.x.116:2375\r\n  └ Status: Healthy\r\n  └ Containers: 138\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-117: x.x.x.117:2375\r\n  └ Status: Healthy\r\n  └ Containers: 143\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-123: x.x.x.123:2375\r\n  └ Status: Healthy\r\n  └ Containers: 130\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-124: x.x.x.124:2375\r\n  └ Status: Healthy\r\n  └ Containers: 165\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 69.5 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-125: x.x.x.125:2375\r\n  └ Status: Pending\r\n  └ Containers: 0\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 0 B / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:22:46Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-126: x.x.x.126:2375\r\n  └ Status: Healthy\r\n  └ Containers: 112\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 72.5 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-127: x.x.x.127:2375\r\n  └ Status: Healthy\r\n  └ Containers: 134\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\n worker-x-x-x-129: x.x.x.129:2375\r\n  └ Status: Healthy\r\n  └ Containers: 138\r\n  └ Reserved CPUs: 0 / 42\r\n  └ Reserved Memory: 70 GiB / 132.2 GiB\r\n  └ Labels: executiondriver=native-0.2, kernelversion=4.4.1-coreos, operatingsystem=CoreOS 962.0.0 (Coeur Rouge), storagedriver=overlay\r\n  └ Error: (none)\r\n  └ UpdatedAt: 2016-04-27T11:23:03Z\r\n  └ ServerVersion: 1.10.1\r\nPlugins:\r\n Volume:\r\n Network:\r\nKernel Version: 4.4.1-coreos\r\nOperating System: linux\r\nArchitecture: amd64\r\nCPUs: 462\r\nTotal Memory: 1.42 TiB\r\nName: worker-x-x-x-121\r\nExperimental: true\r\n```\r\n\r\nswarm version:\r\n```\r\nsystem # /opt/bin/swarm --version\r\nswarm version 1.2.0 ()\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/swarm/issues/comments/217504009","html_url":"https://github.com/docker/swarm/issues/2178#issuecomment-217504009","issue_url":"https://api.github.com/repos/docker/swarm/issues/2178","id":217504009,"user":{"login":"dongluochen","id":14985073,"avatar_url":"https://avatars.githubusercontent.com/u/14985073?v=3","gravatar_id":"","url":"https://api.github.com/users/dongluochen","html_url":"https://github.com/dongluochen","followers_url":"https://api.github.com/users/dongluochen/followers","following_url":"https://api.github.com/users/dongluochen/following{/other_user}","gists_url":"https://api.github.com/users/dongluochen/gists{/gist_id}","starred_url":"https://api.github.com/users/dongluochen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongluochen/subscriptions","organizations_url":"https://api.github.com/users/dongluochen/orgs","repos_url":"https://api.github.com/users/dongluochen/repos","events_url":"https://api.github.com/users/dongluochen/events{/privacy}","received_events_url":"https://api.github.com/users/dongluochen/received_events","type":"User","site_admin":false},"created_at":"2016-05-06T17:21:11Z","updated_at":"2016-05-06T17:21:11Z","body":"@shenshouer Besides nishant's questions, do you get call stack of all goroutines from the panic? We want to take a look where the goroutines are from. If you don't get that, there is a debug option to get call stack from a running manager.  Can you start swarm manager with `debug` option (different from loglevel) like `swarm --debug manage ...`? You can use the following command to print goroutine stack `curl swarm-manager:port/debug/pprof/goroutine?debug=1`. It'd be very helpful to get the call stack after you apply your workload on the system. "}}