{"action":"created","issue":{"url":"https://api.github.com/repos/docker/for-azure/issues/53","repository_url":"https://api.github.com/repos/docker/for-azure","labels_url":"https://api.github.com/repos/docker/for-azure/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/docker/for-azure/issues/53/comments","events_url":"https://api.github.com/repos/docker/for-azure/issues/53/events","html_url":"https://github.com/docker/for-azure/issues/53","id":302086188,"number":53,"title":"ghost container with cloudstor-metadata no such file or directory","user":{"login":"djeeg","id":4269005,"avatar_url":"https://avatars1.githubusercontent.com/u/4269005?v=4","gravatar_id":"","url":"https://api.github.com/users/djeeg","html_url":"https://github.com/djeeg","followers_url":"https://api.github.com/users/djeeg/followers","following_url":"https://api.github.com/users/djeeg/following{/other_user}","gists_url":"https://api.github.com/users/djeeg/gists{/gist_id}","starred_url":"https://api.github.com/users/djeeg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djeeg/subscriptions","organizations_url":"https://api.github.com/users/djeeg/orgs","repos_url":"https://api.github.com/users/djeeg/repos","events_url":"https://api.github.com/users/djeeg/events{/privacy}","received_events_url":"https://api.github.com/users/djeeg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-03-04T13:34:34Z","updated_at":"2018-03-14T21:53:42Z","closed_at":null,"author_association":"NONE","body":"Hi, \r\n\r\nYesterday I went ahead and performed a full swarm rebuild of my azure swarm\r\nThe migration from 17.09-ish to 17.12.0 seemed to go smoothly\r\nI got everything working correctly before knocking off of the day\r\n\r\nThis morning I found the application isnt working and see lots of errors from a redis service\r\n\r\n> MISCONF Errors writing to the AOF file: No such file or directory\r\n\r\nFirst I try restarting the service\r\n\r\n```\r\n$ docker service scale data_redis=0\r\ndata_redis scaled to 0\r\noverall progress: 0 out of 0 tasks\r\nverify: Service converged\r\n```\r\n\r\nHowever the service doesnt scale down\r\n\r\n```\r\n$ docker stack services data\r\nID                  NAME                       MODE                REPLICAS            IMAGE                                                             PORTS\r\nmfj325xz8rj4        data_rabbitmq         replicated          1/1                 rabbitmq:3.7.3-management\r\nydqo5bc9xrrs        data_redis            replicated          1/0                 redis:4.0.6:latest\r\n```\r\n\r\nI can still see the container running\r\n\r\n```\r\n$ docker ps | grep redis\r\neff366c1acb0        redis:4.0.6:latest        \"docker-entrypoint.s…\"   18 hours ago        Up 18 hours (healthy)   6379/tcp, 7000/tcp                         data_redis.1.pfma6polo81rpxx9cxoz4pqt5\r\n```\r\n\r\nI try removing the stack completely, however the container is still running\r\n\r\n```\r\n$ docker stack rm data\r\nRemoving service data_rabbitmq\r\nRemoving service data_redis\r\n```\r\n\r\nNext I try redeploying the stack\r\n\r\n```\r\n$ docker stack services data\r\nID                  NAME                       MODE                REPLICAS            IMAGE                                                             PORTS\r\nwg4uvqd3c4z8        data_rabbitmq         replicated          1/1                 rabbitmq:3.7.3-management\r\nzaxwrx318mtr        data_redis            replicated          1/1                 redis:4.0.6:latest\r\n```\r\n\r\nSeems to bring the application back to life, though i now have another ghost container\r\n\r\n```\r\n$ docker ps | grep redis\r\n405852b52bf1        redis:4.0.6:latest        \"docker-entrypoint.s…\"   6 minutes ago        Up 6 minutes (healthy)   6379/tcp, 7000/tcp                         data_redis.1.qy87hpzmfiaz08f7f6b4g17lx\r\neff366c1acb0        redis:4.0.6:latest        \"docker-entrypoint.s…\"   18 hours ago         Up 18 hours (healthy)    6379/tcp, 7000/tcp                         data_redis.1.pfma6polo81rpxx9cxoz4pqt5\r\n```\r\n\r\nTry to end the container, command hangs the console\r\n\r\n`$ docker kill eff366c1acb0`\r\n\r\nDo some digging into docker logs, these seems related to the redis error\r\n[could not fetch metadata: cannot read metadata: open /mnt/cloudstor/cloudstor-metadata]\r\n[no such file or directory]\r\n\r\n```\r\nMar  4 12:32:27 moby root: time=\"2018-03-04T12:32:27.808280515Z\" level=error msg=\"plugin not found\"\r\nMar  4 12:32:27 moby root: time=\"2018-03-04T12:32:27.808298501Z\" level=error msg=\"plugin not found\"\r\nMar  4 12:32:27 moby root: time=\"2018-03-04T12:32:27.808307994Z\" level=error msg=\"plugin not found\"\r\nMar  4 12:32:27 moby root: time=\"2018-03-04T12:32:27Z\" level=error msg=\"time=\\\"2018-03-04T12:32:27Z\\\" level=debug msg=\\\"request accepted\\\" name=...operation=get \" plugin=...\r\nMar  4 12:32:27 moby root: time=\"2018-03-04T12:32:27Z\" level=error msg=\"time=\\\"2018-03-04T12:32:27Z\\\" level=error msg=\\\"could not fetch metadata: cannot read metadata: open /mnt/cloudstor/cloudstor-metadata/...: no such file or directory\\\" name=...operation=get \" plugin=...\r\nMar  4 12:32:27 moby root: time=\"2018-03-04T12:32:27Z\" level=error msg=\"time=\\\"2018-03-04T12:32:27Z\\\" level=error msg=\\\"could not fetch metadata: cannot read metadata: open /mnt/cloudstor/cloudstor-metadata/...: no such file or directory\\\" name=...operation=get \" plugin=...\r\nMar  4 12:32:27 moby root: time=\"2018-03-04T12:32:27Z\" level=error msg=\"time=\\\"2018-03-04T12:32:27Z\\\" level=error msg=\\\"could not fetch metadata: cannot read metadata: open /mnt/cloudstor/cloudstor-metadata/...: no such file or directory\\\" name=...operation=get \" plugin=...\r\nMar  4 12:32:32 moby root: time=\"2018-03-04T12:32:32.794547978Z\" level=error msg=\"Failed to deserialize netlink ndmsg: Link not found\"\r\n```\r\n\r\nI see that [Failed to deserialize netlink ndmsg: Link not found] has been fixed with https://github.com/docker/libnetwork/issues/2045, could it be related to the cloudstor failure?\r\n\r\nCurrent stack config (though I will change this now to replace cloudstor with a local volume)\r\n\r\n\r\n```\r\nservices:\r\n\r\n  redis:\r\n    image: redis:4.0.6\r\n    networks:\r\n      - network1\r\n\t  - network2\r\n\t  - network3\r\n    volumes:\r\n      - volredis:/data\r\n    secrets:\r\n     - secret1\r\n    configs:\r\n      - source: redis_conf\r\n        target: /redis-conf/redis.conf\r\n        uid: '0'\r\n        gid: '0'\r\n        mode: 0444\r\n    logging:\r\n\t  driver: json-file\r\n\t  options:\r\n\t\tmax-size: \"50m\"\r\n\t\tmax-file: \"2\"\r\n    deploy:\r\n      placement:\r\n        constraints:\r\n          - node.role == manager\r\n      resources:\r\n\t\t  limits:\r\n\t\t\tcpus: '0.25'\r\n\t\t\tmemory: 1000M\r\n\t\t  reservations:\r\n\t\t\t#cpus: '0.25'\r\n\t\t\tmemory: 200M\r\n\r\n\r\nvolumes:\r\n\r\n  volredis:\r\n    name: '{{index .Service.Labels \"com.docker.stack.namespace\"}}_volredis_{{.Task.Slot}}'\r\n    driver: cloudstor:azure\r\n```\r\n\r\nAnd redis config\r\n\r\n```\r\nbind 0.0.0.0\r\nport 7000\r\nappendonly yes\r\nprotected-mode no\r\ndatabases 3\r\n```\r\n\r\ndocker info\r\n\r\n\r\n```\r\nContainers: 82\r\n Running: 33\r\n Paused: 0\r\n Stopped: 49\r\nImages: 24\r\nServer Version: 17.12.0-ce\r\nStorage Driver: overlay2\r\n Backing Filesystem: extfs\r\n Supports d_type: true\r\n Native Overlay Diff: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: active\r\n NodeID: \r\n Is Manager: true\r\n ClusterID: \r\n Managers: 1\r\n Nodes: 2\r\n Orchestration:\r\n  Task History Retention Limit: 5\r\n Raft:\r\n  Snapshot Interval: 10000\r\n  Number of Old Snapshots to Retain: 0\r\n  Heartbeat Tick: 1\r\n  Election Tick: 3\r\n Dispatcher:\r\n  Heartbeat Period: 5 seconds\r\n CA Configuration:\r\n  Expiry Duration: 3 months\r\n  Force Rotate: 0\r\n Autolock Managers: false\r\n Root Rotation In Progress: false\r\n Node Address: 10.0.0.4\r\n Manager Addresses:\r\n  10.0.0.4:2377\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 89623f28b87a6004d4b785663257362d1658a729\r\nrunc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f\r\ninit version: 949e6fa\r\nSecurity Options:\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.9.75-moby\r\nOperating System: Alpine Linux v3.5\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 6.784GiB\r\nName: swarm-manager000000\r\nID: \r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 325\r\n Goroutines: 600\r\n System Time: 2018-03-04T13:26:38.411040014Z\r\n EventsListeners: 27\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/docker/for-azure/issues/comments/373187559","html_url":"https://github.com/docker/for-azure/issues/53#issuecomment-373187559","issue_url":"https://api.github.com/repos/docker/for-azure/issues/53","id":373187559,"user":{"login":"FrenchBen","id":3965324,"avatar_url":"https://avatars0.githubusercontent.com/u/3965324?v=4","gravatar_id":"","url":"https://api.github.com/users/FrenchBen","html_url":"https://github.com/FrenchBen","followers_url":"https://api.github.com/users/FrenchBen/followers","following_url":"https://api.github.com/users/FrenchBen/following{/other_user}","gists_url":"https://api.github.com/users/FrenchBen/gists{/gist_id}","starred_url":"https://api.github.com/users/FrenchBen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FrenchBen/subscriptions","organizations_url":"https://api.github.com/users/FrenchBen/orgs","repos_url":"https://api.github.com/users/FrenchBen/repos","events_url":"https://api.github.com/users/FrenchBen/events{/privacy}","received_events_url":"https://api.github.com/users/FrenchBen/received_events","type":"User","site_admin":false},"created_at":"2018-03-14T21:53:42Z","updated_at":"2018-03-14T21:53:42Z","author_association":"CONTRIBUTOR","body":"is redis crashing due to the read/write on the cloudstor volume, or other errors are popping up?"}}
{"actor":{"display_login":"FrenchBen"}}