{"action":"created","issue":{"url":"https://api.github.com/repos/cloudstax/firecamp/issues/41","repository_url":"https://api.github.com/repos/cloudstax/firecamp","labels_url":"https://api.github.com/repos/cloudstax/firecamp/issues/41/labels{/name}","comments_url":"https://api.github.com/repos/cloudstax/firecamp/issues/41/comments","events_url":"https://api.github.com/repos/cloudstax/firecamp/issues/41/events","html_url":"https://github.com/cloudstax/firecamp/issues/41","id":301792248,"number":41,"title":"Tasks can't be started on an EC2 instance","user":{"login":"jazzl0ver","id":2444175,"avatar_url":"https://avatars2.githubusercontent.com/u/2444175?v=4","gravatar_id":"","url":"https://api.github.com/users/jazzl0ver","html_url":"https://github.com/jazzl0ver","followers_url":"https://api.github.com/users/jazzl0ver/followers","following_url":"https://api.github.com/users/jazzl0ver/following{/other_user}","gists_url":"https://api.github.com/users/jazzl0ver/gists{/gist_id}","starred_url":"https://api.github.com/users/jazzl0ver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jazzl0ver/subscriptions","organizations_url":"https://api.github.com/users/jazzl0ver/orgs","repos_url":"https://api.github.com/users/jazzl0ver/repos","events_url":"https://api.github.com/users/jazzl0ver/events{/privacy}","received_events_url":"https://api.github.com/users/jazzl0ver/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-03-02T14:53:52Z","updated_at":"2018-03-05T17:11:02Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"There was an issue with one of EC2 instances, so I've terminated it and the ASG has started new one. For some reason, the containers are not starting up on the new instance. They fail with (from /var/log/docker):\r\n```\r\ntime=\"2018-03-02T10:07:55Z\" level=info msg=\"2018/03/02 10:07:55 http: panic serving @: runtime error: invalid memory address or nil pointer dereference\" plugin=3c95129f659d2d162550065c4200980c15d8d2ce25c002c9f01f96c84f3ea636\r\ntime=\"2018-03-02T10:07:55.565099089Z\" level=warning msg=\"Unable to connect to plugin: /run/docker/plugins/3c95129f659d2d162550065c4200980c15d8d2ce25c002c9f01f96c84f3ea636/firecampvol.sock/VolumeDriver.Mount: Post http://%2Frun%2Fdocker%2Fplugins%2F3c95129f659d2d162550065c4200980c15d8d2ce25c002c9f01f96c84f3ea636%2Ffirecampvol.sock/VolumeDriver.Mount: EOF, retrying in 1s\"\r\n```\r\nSo, no volumes are mounted.\r\n```\r\n[root@ip-172-22-2-212 log]# docker ps\r\nCONTAINER ID        IMAGE                                        COMMAND             CREATED             STATUS              PORTS               NAMES\r\n634ef2c0ea2f        cloudstax/firecamp-amazon-ecs-agent:latest   \"/agent\"            5 hours ago         Up 5 hours                              ecs-agent\r\n[root@ip-172-22-2-212 log]# docker plugin ls\r\nID                  NAME                               DESCRIPTION                                     ENABLED\r\n3c95129f659d        cloudstax/firecamp-volume:latest   firecamp volume plugin for docker               true\r\n656134559eb0        cloudstax/firecamp-log:latest      firecamp log plugin for docker: consume lo...   true\r\n```\r\nThe only thing I did against this cluster recently was killing firecamp-manageserver task to make it updated to the latest.\r\nOther two cluster nodes work w/o issues.\r\nThe only difference I see is the agent version:\r\n```\r\ngrep Agent /var/log/firecamp/firecamp-dockervolume.INFO\r\n```\r\n\"Amazon ECS Agent - v1.16.2 (*55b7b5f)\" - at the new (non-working) instance\r\n\"Amazon ECS Agent - v1.16.0 (*e24ae08)\" - at working instances\r\n\r\nPlease, help me to figure that out!"},"comment":{"url":"https://api.github.com/repos/cloudstax/firecamp/issues/comments/370491404","html_url":"https://github.com/cloudstax/firecamp/issues/41#issuecomment-370491404","issue_url":"https://api.github.com/repos/cloudstax/firecamp/issues/41","id":370491404,"user":{"login":"JuniusLuo","id":28828038,"avatar_url":"https://avatars2.githubusercontent.com/u/28828038?v=4","gravatar_id":"","url":"https://api.github.com/users/JuniusLuo","html_url":"https://github.com/JuniusLuo","followers_url":"https://api.github.com/users/JuniusLuo/followers","following_url":"https://api.github.com/users/JuniusLuo/following{/other_user}","gists_url":"https://api.github.com/users/JuniusLuo/gists{/gist_id}","starred_url":"https://api.github.com/users/JuniusLuo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuniusLuo/subscriptions","organizations_url":"https://api.github.com/users/JuniusLuo/orgs","repos_url":"https://api.github.com/users/JuniusLuo/repos","events_url":"https://api.github.com/users/JuniusLuo/events{/privacy}","received_events_url":"https://api.github.com/users/JuniusLuo/received_events","type":"User","site_admin":false},"created_at":"2018-03-05T17:11:02Z","updated_at":"2018-03-05T17:11:02Z","author_association":"CONTRIBUTOR","body":"Your are right. The latest manageserver is not able to delete the service. It will hit the same issue when listing all service members.\r\n\r\nUsing the downgraded firecamp-manager would be able to delete the service. For the deletion failure, could you please post the manageserver log?"}}
{"actor":{"display_login":"JuniusLuo"}}