{"action":"created","issue":{"url":"https://api.github.com/repos/docker/swarm/issues/2108","repository_url":"https://api.github.com/repos/docker/swarm","labels_url":"https://api.github.com/repos/docker/swarm/issues/2108/labels{/name}","comments_url":"https://api.github.com/repos/docker/swarm/issues/2108/comments","events_url":"https://api.github.com/repos/docker/swarm/issues/2108/events","html_url":"https://github.com/docker/swarm/issues/2108","id":147431677,"number":2108,"title":"docker ps inconsistency: master vs node","user":{"login":"Garreat","id":10948016,"avatar_url":"https://avatars.githubusercontent.com/u/10948016?v=3","gravatar_id":"","url":"https://api.github.com/users/Garreat","html_url":"https://github.com/Garreat","followers_url":"https://api.github.com/users/Garreat/followers","following_url":"https://api.github.com/users/Garreat/following{/other_user}","gists_url":"https://api.github.com/users/Garreat/gists{/gist_id}","starred_url":"https://api.github.com/users/Garreat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Garreat/subscriptions","organizations_url":"https://api.github.com/users/Garreat/orgs","repos_url":"https://api.github.com/users/Garreat/repos","events_url":"https://api.github.com/users/Garreat/events{/privacy}","received_events_url":"https://api.github.com/users/Garreat/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/swarm/labels/impact/api","name":"impact/api","color":"eb6420"},{"url":"https://api.github.com/repos/docker/swarm/labels/impact/cli","name":"impact/cli","color":"eb6420"},{"url":"https://api.github.com/repos/docker/swarm/labels/kind/bug","name":"kind/bug","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":25,"created_at":"2016-04-11T13:35:58Z","updated_at":"2016-04-27T02:25:25Z","closed_at":null,"body":"Greetings Swarm community!\r\nThis issue seems somewhat related to:\r\n[https://github.com/shipyard/shipyard/issues/728](https://github.com/shipyard/shipyard/issues/728)\r\n[https://github.com/docker/swarm/issues/1468](https://github.com/docker/swarm/issues/1468)\r\n\r\n**Output of `docker version`:**\r\n```\r\nClient:\r\n Version:      1.10.2\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   c3959b1\r\n Built:        Mon Feb 22 16:16:33 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.10.2\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   c3959b1\r\n Built:        Mon Feb 22 16:16:33 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n```\r\nContainers: 4\r\n Running: 4\r\n Paused: 0\r\n Stopped: 0\r\nImages: 4\r\nServer Version: 1.10.2\r\nStorage Driver: devicemapper\r\n Pool Name: docker-253:3-17700697-pool\r\n Pool Blocksize: 65.54 kB\r\n Base Device Size: 10.74 GB\r\n Backing Filesystem: xfs\r\n Data file: /dev/loop0\r\n Metadata file: /dev/loop1\r\n Data Space Used: 1.601 GB\r\n Data Space Total: 107.4 GB\r\n Data Space Available: 22.19 GB\r\n Metadata Space Used: 3.092 MB\r\n Metadata Space Total: 2.147 GB\r\n Metadata Space Available: 2.144 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Deferred Deletion Enabled: false\r\n Deferred Deleted Device Count: 0\r\n Data loop file: /var/lib/docker/devicemapper/devicemapper/data\r\n WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.\r\n Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r\n Library Version: 1.02.107-RHEL7 (2015-12-01)\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nPlugins: \r\n Volume: local\r\n Network: overlay bridge null host\r\nKernel Version: 3.10.0-327.10.1.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 6\r\nTotal Memory: 7.797 GiB\r\nName: n1\r\nID: WP23:YQON:7YTL:WFXI:WFWM:JL65:7GOA:FUKP:PISL:3IQ3:S3AH:VPBZ\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\nLabels:\r\n role=node\r\nCluster store: consul://127.0.0.1:8500\r\nCluster advertise: x.x.x.x:2375\r\n```\r\n\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nIn my environment, all Swarm members are VSphere VMs within a single VLAN.\r\n\r\n\r\n**Steps to reproduce the issue:**\r\n1. Assuming you already have a Swarm environment...\r\n2. Start a container.\r\n3. Gracefully stop the container from inside. Don't use docker commands, simply lead container's PID 1 to a natural end.\r\n\r\n\r\n**Describe the results you received:**\r\nSwarm manager shows the container still running (incorrect):\r\n```\r\ndockkv1~# docker -H :3375 ps\r\n(...)\r\n485a2805a838        env-test        \"/init /usr/bin/..\"   13 hours ago        Up 13 hours        dockn1/test8\r\n```\r\nHowever, the node shows the container exited (correct):\r\n```\r\ndockn1~# docker ps -a\r\n(...)\r\n485a2805a838        env-test        \"/init /usr/bin/..\"   13 hours ago        Exited (0) 13 hours ago        test8\r\n```\r\n\r\n\r\n**Describe the results you expected:**\r\nSwarm manager should correctly report the container state.\r\nWith this issue, it's not hard to imagine an orchestration scenario failing.\r\n\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\nI use s6-overlay in the test container ([https://github.com/just-containers/s6-overlay](https://github.com/just-containers/s6-overlay)). BUT I have just reproduced this scenario with nginx container, using `nsenter` to go inside and then `nginx -s quit`. Result is the same: exited on the node, running on the master.\r\nThe containers are definitely stopped:\r\n```\r\ndockkv1~# docker -H :3375 stats ngx0 test8\r\nCONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O\r\nngx0                0.00%               0 B / 0 B           0.00%               0 B / 0 B           0 B / 0 B\r\ntest8               0.00%               0 B / 0 B           0.00%               0 B / 0 B           0 B / 0 B\r\n```\r\nThe above containers left their overlay networks gracefully.\r\nSimple `rm` command (without `-f`) is enough to fix this - despite containers being shown running.\r\nThe situation persists until an administrative action is taken manually."},"comment":{"url":"https://api.github.com/repos/docker/swarm/issues/comments/214950144","html_url":"https://github.com/docker/swarm/issues/2108#issuecomment-214950144","issue_url":"https://api.github.com/repos/docker/swarm/issues/2108","id":214950144,"user":{"login":"chrodriguez","id":1460882,"avatar_url":"https://avatars.githubusercontent.com/u/1460882?v=3","gravatar_id":"","url":"https://api.github.com/users/chrodriguez","html_url":"https://github.com/chrodriguez","followers_url":"https://api.github.com/users/chrodriguez/followers","following_url":"https://api.github.com/users/chrodriguez/following{/other_user}","gists_url":"https://api.github.com/users/chrodriguez/gists{/gist_id}","starred_url":"https://api.github.com/users/chrodriguez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrodriguez/subscriptions","organizations_url":"https://api.github.com/users/chrodriguez/orgs","repos_url":"https://api.github.com/users/chrodriguez/repos","events_url":"https://api.github.com/users/chrodriguez/events{/privacy}","received_events_url":"https://api.github.com/users/chrodriguez/received_events","type":"User","site_admin":false},"created_at":"2016-04-27T02:25:25Z","updated_at":"2016-04-27T02:25:25Z","body":"Hi everyone! I've recently installed a swarm cluster and evidenced some kind of the behavior described by this issue, but in a different way.\r\n\r\nFirst I will describe my infraestructure:\r\n\r\n1. Consul HA + Swarm manager HA\r\n  1. SERVER 1: consul server + swam manager\r\n  2. SERVER 2: consul server + swarm manager \r\n 3. SERVER 3: consul server + swarm manager  \r\n2. Swarm nodes\r\n  1. SERVER 1: swarm node\r\n  2. SERVER 2: swarm node\r\n\r\nEach swarm node runs **registrator** and every new docker container run inside a swarm manager were run with an `always restart policy`\r\n\r\nI've started 6 nginx containers this way and reboot one of the swarm nodes server (where there were at least one container running), and then `docker -H :3376 ps ` will consider that rebooted server containers are not running, but `docker ps` in that swarm node reports container is running. Moreover, those containers shows a running status when they are inspected with `docker -H :3376 inspect xxxx`\r\n\r\nI don't know how this issue may impact a production infrastructure, but is not trivial to read ps output when reports invalid information from the swarm cluster.\r\n\r\nI hope my description helps solving this issue, thanks in advance!"}}