{"action":"created","issue":{"url":"https://api.github.com/repos/uswitch/kiam/issues/46","repository_url":"https://api.github.com/repos/uswitch/kiam","labels_url":"https://api.github.com/repos/uswitch/kiam/issues/46/labels{/name}","comments_url":"https://api.github.com/repos/uswitch/kiam/issues/46/comments","events_url":"https://api.github.com/repos/uswitch/kiam/issues/46/events","html_url":"https://github.com/uswitch/kiam/issues/46","id":305442632,"number":46,"title":"unable to assign pod the iam role ","user":{"login":"tasdikrahman","id":4672518,"avatar_url":"https://avatars1.githubusercontent.com/u/4672518?v=4","gravatar_id":"","url":"https://api.github.com/users/tasdikrahman","html_url":"https://github.com/tasdikrahman","followers_url":"https://api.github.com/users/tasdikrahman/followers","following_url":"https://api.github.com/users/tasdikrahman/following{/other_user}","gists_url":"https://api.github.com/users/tasdikrahman/gists{/gist_id}","starred_url":"https://api.github.com/users/tasdikrahman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tasdikrahman/subscriptions","organizations_url":"https://api.github.com/users/tasdikrahman/orgs","repos_url":"https://api.github.com/users/tasdikrahman/repos","events_url":"https://api.github.com/users/tasdikrahman/events{/privacy}","received_events_url":"https://api.github.com/users/tasdikrahman/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-03-15T07:32:19Z","updated_at":"2018-03-15T10:30:48Z","closed_at":"2018-03-15T10:30:48Z","author_association":"NONE","body":"checked the kiam-agent logs on the node where the pod (which was to be assigned the iam role) was scheduled which look like\r\n\r\n```\r\n{\"addr\":\"10.2.40.2:34938\",\"headers\":{},\"level\":\"info\",\"method\":\"GET\",\"msg\":\"processed request\",\"path\":\"/latest/meta-data/iam/security-credentials/\",\"status\":200,\"time\":\"2018-03-15T07:02:32Z\"}\r\n{\"level\":\"warning\",\"msg\":\"error finding role for pod: rpc error: code = Unknown desc = pod not found\",\"pod.ip\":\"10.2.40.2\",\"time\":\"2018-03-15T07:02:32Z\"}\r\n{\"addr\":\"10.2.40.2:34940\",\"level\":\"error\",\"method\":\"GET\",\"msg\":\"error processing request: error checking assume role permissions: rpc error: code = Unknown desc = pod not found\",\"path\":\"/latest/meta-data/iam/security-credentials/my-app-iam-role\",\"status\":500,\"time\":\"2018-03-15T07:02:32Z\"}\r\n{\"addr\":\"10.2.40.2:34940\",\"headers\":{\"Content-Type\":[\"text/plain; charset=utf-8\"],\"X-Content-Type-Options\":[\"nosniff\"]},\"level\":\"info\",\"method\":\"GET\",\"msg\":\"processed request\",\"path\":\"/latest/meta-data/iam/security-credentials/my-app-iam-role\",\"status\":500,\"time\":\"2018-03-15T07:02:32Z\"}\r\n{\"addr\":\"10.2.40.2:34942\",\"headers\":{},\"level\":\"info\",\"method\":\"GET\",\"msg\":\"processed request\",\"path\":\"/latest/meta-data/iam/security-credentials/\",\"status\":200,\"time\":\"2018-03-15T07:02:33Z\"}\r\n{\"level\":\"warning\",\"msg\":\"error finding role for pod: rpc error: code = Unknown desc = pod not found\",\"pod.ip\":\"10.2.40.2\",\"time\":\"2018-03-15T07:02:33Z\"}\r\n{\"addr\":\"10.2.40.2:34944\",\"level\":\"error\",\"method\":\"GET\",\"msg\":\"error processing request: error checking assume role permissions: rpc error: code = Unknown desc = pod not found\",\"path\":\"/latest/meta-data/iam/security-credentials/my-app-iam-role\",\"status\":500,\"time\":\"2018-03-15T07:02:33Z\"}\r\n{\"addr\":\"10.2.40.2:34944\",\"headers\":{\"Content-Type\":[\"text/plain; charset=utf-8\"],\"X-Content-Type-Options\":[\"nosniff\"]},\"level\":\"info\",\"method\":\"GET\",\"msg\":\"processed request\",\"path\":\"/latest/meta-data/iam/security-credentials/my-app-iam-role\",\"status\":500,\"time\":\"2018-03-15T07:02:33Z\"}\r\n{\"addr\":\"10.2.40.2:34946\",\"headers\":{},\"level\":\"info\",\"method\":\"GET\",\"msg\":\"processed request\",\"path\":\"/latest/meta-data/iam/security-credentials/\",\"status\":200,\"time\":\"2018-03-15T07:02:33Z\"}\r\n{\"level\":\"warning\",\"msg\":\"error finding role for pod: rpc error: code = Unknown desc = pod not found\",\"pod.ip\":\"10.2.40.2\",\"time\":\"2018-03-15T07:02:33Z\"}\r\n{\"addr\":\"10.2.40.2:34948\",\"level\":\"error\",\"method\":\"GET\",\"msg\":\"error processing request: error checking assume role permissions: rpc error: code = Unknown desc = pod not found\",\"path\":\"/latest/meta-data/iam/security-credentials/my-app-iam-role\",\"status\":500,\"time\":\"2018-03-15T07:02:33Z\"}\r\n{\"addr\":\"10.2.40.2:34948\",\"headers\":{\"Content-Type\":[\"text/plain; charset=utf-8\"],\"X-Content-Type-Options\":[\"nosniff\"]},\"level\":\"info\",\"method\":\"GET\",\"msg\":\"processed request\",\"path\":\"/latest/meta-data/iam/security-credentials/my-app-iam-role\",\"status\":500,\"time\":\"2018-03-15T07:02:33Z\"}\r\n```\r\n\r\nrunning `uswitch/kiam:v2.4` on both the agent and the server. \r\n\r\nThe namespace where the pod is scheduled has the annotation \r\n```\r\n  annotations:\r\n    iam.amazonaws.com/permitted: .*\r\n```\r\nas stated in the docs\r\n\r\nalong with \r\n\r\n```\r\n{\r\n  \"Version\": \"2012-10-17\",\r\n  \"Statement\": [\r\n    {\r\n      \"Sid\": \"\",\r\n      \"Effect\": \"Allow\",\r\n      \"Principal\": {\r\n        \"Service\": \"ec2.amazonaws.com\"\r\n      },\r\n      \"Action\": \"sts:AssumeRole\"\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\nin the trust relationships on the iam-role picked up the node where the pods are being scheduled. \r\n\r\nNot sure if it's related, but we recently upgraded k8s from 1.8.4 to 1.8.9, but I guess that shouldn't be the problem. \r\n"},"comment":{"url":"https://api.github.com/repos/uswitch/kiam/issues/comments/373330662","html_url":"https://github.com/uswitch/kiam/issues/46#issuecomment-373330662","issue_url":"https://api.github.com/repos/uswitch/kiam/issues/46","id":373330662,"user":{"login":"pingles","id":1150,"avatar_url":"https://avatars0.githubusercontent.com/u/1150?v=4","gravatar_id":"","url":"https://api.github.com/users/pingles","html_url":"https://github.com/pingles","followers_url":"https://api.github.com/users/pingles/followers","following_url":"https://api.github.com/users/pingles/following{/other_user}","gists_url":"https://api.github.com/users/pingles/gists{/gist_id}","starred_url":"https://api.github.com/users/pingles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pingles/subscriptions","organizations_url":"https://api.github.com/users/pingles/orgs","repos_url":"https://api.github.com/users/pingles/repos","events_url":"https://api.github.com/users/pingles/events{/privacy}","received_events_url":"https://api.github.com/users/pingles/received_events","type":"User","site_admin":false},"created_at":"2018-03-15T10:30:48Z","updated_at":"2018-03-15T10:30:48Z","author_association":"CONTRIBUTOR","body":"Yeah sorry, without the server logs its difficult to know what the problem is. I'm going to close this for now but please reopen with if it happens again with as much log data as you can capture please.\r\n\r\nThanks!"}}
{"actor":{"display_login":"pingles"}}