{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/37073","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/37073/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/37073/comments","events_url":"https://api.github.com/repos/moby/moby/issues/37073/events","html_url":"https://github.com/moby/moby/issues/37073","id":323434420,"number":37073,"title":"node null not found error, service stuck with new state","user":{"login":"bastiflew","id":16416882,"avatar_url":"https://avatars0.githubusercontent.com/u/16416882?v=4","gravatar_id":"","url":"https://api.github.com/users/bastiflew","html_url":"https://github.com/bastiflew","followers_url":"https://api.github.com/users/bastiflew/followers","following_url":"https://api.github.com/users/bastiflew/following{/other_user}","gists_url":"https://api.github.com/users/bastiflew/gists{/gist_id}","starred_url":"https://api.github.com/users/bastiflew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bastiflew/subscriptions","organizations_url":"https://api.github.com/users/bastiflew/orgs","repos_url":"https://api.github.com/users/bastiflew/repos","events_url":"https://api.github.com/users/bastiflew/events{/privacy}","received_events_url":"https://api.github.com/users/bastiflew/received_events","type":"User","site_admin":false},"labels":[{"id":392678049,"url":"https://api.github.com/repos/moby/moby/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-16T01:04:58Z","updated_at":"2018-05-24T20:29:32Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\n\r\nSince today, I have this issue that I can't fix with docker swarm.\r\nMost of the replicas can't start, they stay into \"new\" state without a node attributed\r\n\r\n```ID                  NAME                                     IMAGE                                         NODE                                 DESIRED STATE       CURRENT STATE                ERROR               PORTS\r\nj33jnn1pa048        backend_preview-internal-backend.1       quay.acme.com/smart/backend.internal:1.3.23   swarm-worker-02.smart.acme.org       Running             Running about an hour ago\r\nc229dwboskik         \\_ backend_preview-internal-backend.1   quay.acme.com/smart/backend.internal:1.3.23   swarm-manager-02.smart.acme.org      Shutdown            Shutdown about an hour ago\r\ni0jqy75m4bjg        backend_preview-internal-backend.2       quay.acme.com/smart/backend.internal:1.3.23   swarm-manager-01.smart.acme.org      Running             Running 8 minutes ago\r\n0ui98kdrpxp8        backend_preview-internal-backend.3       quay.acme.com/smart/backend.internal:1.3.23                                        Running             New 9 minutes ago\r\nz3tkqvfyka23        backend_preview-internal-backend.4       quay.acme.com/smart/backend.internal:1.3.23                                        Running             New 9 minutes ago\r\nkpamd2zxgmc2        backend_preview-internal-backend.5       quay.acme.com/smart/backend.internal:1.3.23                                        Running             New 9 minutes ago\r\nildzzir56340        backend_preview-internal-backend.6       quay.acme.com/smart/backend.internal:1.3.23                                        Running             New 9 minutes ago\r\n```\r\n\r\nOn the 3 managers with `journalctl -fu docker.service` I have this log continuously:\r\n```\r\nMay 16 00:57:36 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:36.199069535Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:41 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:41.275237547Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:41 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:41.338913421Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:41 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:41.414829123Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:46 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:46.487962133Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:46 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:46.549313053Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:46 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:46.621488846Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:51 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:51.692517060Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:51 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:51.753280638Z\" level=error msg=\"Error getting node null: node null not found\"\r\nMay 16 00:57:51 swarm-manager-02.smart.acme.org env[3479]: time=\"2018-05-16T00:57:51.824797787Z\" level=error msg=\"Error getting node null: node null not found\"\r\n```\r\ndocker version:\r\n```\r\nClient:\r\n Version:      18.03.1-ce\r\n API version:  1.37\r\n Go version:   go1.9.4\r\n Git commit:   9ee9f40\r\n Built:        Thu Apr 26 04:27:49 2018\r\n OS/Arch:      linux/amd64\r\n Experimental: false\r\n Orchestrator: swarm\r\n\r\nServer:\r\n Engine:\r\n  Version:      18.03.1-ce\r\n  API version:  1.37 (minimum version 1.12)\r\n  Go version:   go1.9.4\r\n  Git commit:   9ee9f40\r\n  Built:        Thu Apr 26 04:27:49 2018\r\n  OS/Arch:      linux/amd64\r\n  Experimental: true\r\n```\r\n\r\ndocker info:\r\n```\r\nContainers: 35\r\n Running: 35\r\n Paused: 0\r\n Stopped: 0\r\nImages: 31\r\nServer Version: 18.03.1-ce\r\nStorage Driver: overlay2\r\n Backing Filesystem: xfs\r\n Supports d_type: true\r\n Native Overlay Diff: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host ipvlan macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: active\r\n NodeID: l3ku0pkmcdsq1cgx3hnv8a07i\r\n Is Manager: true\r\n ClusterID: wpdw3up06qrulow40cb0cketk\r\n Managers: 3\r\n Nodes: 6\r\n Orchestration:\r\n  Task History Retention Limit: 1\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Number of Old Snapshots to Retain: 0\r\n  Heartbeat Tick: 1\r\n  Election Tick: 3\r\n Dispatcher:\r\n  Heartbeat Period: 5 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n  Force Rotate: 0\r\n Autolock Managers: false\r\n Root Rotation In Progress: false\r\n Node Address: 10.162.128.41\r\n Manager Addresses:\r\n  10.162.128.40:2377\r\n  10.162.128.41:2377\r\n  10.162.128.42:2377\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88\r\nrunc version: 4fc53a81fb7c994640722ac585fa9ca548971871\r\ninit version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)\r\nSecurity Options:\r\n seccomp\r\n  Profile: default\r\n selinux\r\nKernel Version: 4.14.39-coreos\r\nOperating System: Container Linux by CoreOS 1745.2.0 (Rhyolite)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 16\r\nTotal Memory: 29.45GiB\r\nName: swarm-manager-02.smart.acme.org\r\nID: 7KC3:YI4V:DYQZ:KK2K:PA2Q:NVDY:AUDP:MJ2U:I2PD:GLKC:UJJL:YPQL\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\nExperimental: true\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n```\r\n\r\nThanks !"},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/391849063","html_url":"https://github.com/moby/moby/issues/37073#issuecomment-391849063","issue_url":"https://api.github.com/repos/moby/moby/issues/37073","id":391849063,"user":{"login":"bastiflew","id":16416882,"avatar_url":"https://avatars0.githubusercontent.com/u/16416882?v=4","gravatar_id":"","url":"https://api.github.com/users/bastiflew","html_url":"https://github.com/bastiflew","followers_url":"https://api.github.com/users/bastiflew/followers","following_url":"https://api.github.com/users/bastiflew/following{/other_user}","gists_url":"https://api.github.com/users/bastiflew/gists{/gist_id}","starred_url":"https://api.github.com/users/bastiflew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bastiflew/subscriptions","organizations_url":"https://api.github.com/users/bastiflew/orgs","repos_url":"https://api.github.com/users/bastiflew/repos","events_url":"https://api.github.com/users/bastiflew/events{/privacy}","received_events_url":"https://api.github.com/users/bastiflew/received_events","type":"User","site_admin":false},"created_at":"2018-05-24T20:29:32Z","updated_at":"2018-05-24T20:29:32Z","author_association":"NONE","body":"don't care about the message `Error getting node null: node null not found`, it's a script I ran on the server.\r\nI downgrade to docker 17.09.01 for the other issue, seems stable now."}}
{"actor":{"display_login":"bastiflew"}}