{"action":"created","issue":{"url":"https://api.github.com/repos/miracle2k/k8s-snapshots/issues/52","repository_url":"https://api.github.com/repos/miracle2k/k8s-snapshots","labels_url":"https://api.github.com/repos/miracle2k/k8s-snapshots/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/miracle2k/k8s-snapshots/issues/52/comments","events_url":"https://api.github.com/repos/miracle2k/k8s-snapshots/issues/52/events","html_url":"https://github.com/miracle2k/k8s-snapshots/issues/52","id":297933628,"number":52,"title":"Investigate: Load more than the AWS snapshot limit","user":{"login":"shanegibbs","id":2838876,"avatar_url":"https://avatars1.githubusercontent.com/u/2838876?v=4","gravatar_id":"","url":"https://api.github.com/users/shanegibbs","html_url":"https://github.com/shanegibbs","followers_url":"https://api.github.com/users/shanegibbs/followers","following_url":"https://api.github.com/users/shanegibbs/following{/other_user}","gists_url":"https://api.github.com/users/shanegibbs/gists{/gist_id}","starred_url":"https://api.github.com/users/shanegibbs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shanegibbs/subscriptions","organizations_url":"https://api.github.com/users/shanegibbs/orgs","repos_url":"https://api.github.com/users/shanegibbs/repos","events_url":"https://api.github.com/users/shanegibbs/events{/privacy}","received_events_url":"https://api.github.com/users/shanegibbs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-16T22:02:43Z","updated_at":"2018-02-17T23:31:42Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"We are running the `k8s-snapshots:v2.0` container in GKE.\r\n\r\nAfter a while, we start getting a snapshot every minute and our container gets stuck in a crash loop. After deleting all the snapshots and starting again, it works as expected for a while and then starts crash looping again.\r\n\r\nFrom what I can tell, the following happens:\r\n\r\n1. Once we reach a certain threshold of snapshots (turns out to be ~500), snapshots start happening more frequent than has been specified.\r\n2. The more snapshots we have, the worse it gets. We reach a point where we get a new snapshot every minute.\r\n3. We eventually reach our quota limit for snapshots.\r\n4. When `k8s-snapshots` creates a snapshot, the GCP `operation` is flagged as complete, but when the app requests the snapshot info from GCP, it doesn't exist (due to quota limits). This causes a 404 error and crashes the app.\r\n\r\nAfter a bit of digging, I put together a fix that seems to work for us: https://github.com/miracle2k/k8s-snapshots/pull/51"},"comment":{"url":"https://api.github.com/repos/miracle2k/k8s-snapshots/issues/comments/366479417","html_url":"https://github.com/miracle2k/k8s-snapshots/issues/52#issuecomment-366479417","issue_url":"https://api.github.com/repos/miracle2k/k8s-snapshots/issues/52","id":366479417,"user":{"login":"miracle2k","id":13807,"avatar_url":"https://avatars1.githubusercontent.com/u/13807?v=4","gravatar_id":"","url":"https://api.github.com/users/miracle2k","html_url":"https://github.com/miracle2k","followers_url":"https://api.github.com/users/miracle2k/followers","following_url":"https://api.github.com/users/miracle2k/following{/other_user}","gists_url":"https://api.github.com/users/miracle2k/gists{/gist_id}","starred_url":"https://api.github.com/users/miracle2k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miracle2k/subscriptions","organizations_url":"https://api.github.com/users/miracle2k/orgs","repos_url":"https://api.github.com/users/miracle2k/repos","events_url":"https://api.github.com/users/miracle2k/events{/privacy}","received_events_url":"https://api.github.com/users/miracle2k/received_events","type":"User","site_admin":false},"created_at":"2018-02-17T23:31:42Z","updated_at":"2018-02-17T23:31:42Z","author_association":"OWNER","body":"Leaving this open to investigate the related issue on AWS."}}
{"actor":{"display_login":"miracle2k"}}