{"action":"created","issue":{"url":"https://api.github.com/repos/DataBiosphere/leonardo/issues/619","repository_url":"https://api.github.com/repos/DataBiosphere/leonardo","labels_url":"https://api.github.com/repos/DataBiosphere/leonardo/issues/619/labels{/name}","comments_url":"https://api.github.com/repos/DataBiosphere/leonardo/issues/619/comments","events_url":"https://api.github.com/repos/DataBiosphere/leonardo/issues/619/events","html_url":"https://github.com/DataBiosphere/leonardo/issues/619","id":367335839,"node_id":"MDU6SXNzdWUzNjczMzU4Mzk=","number":619,"title":"429 (Too Many Requests) due to \"Quota exceeded for quota metric...\"","user":{"login":"kyuksel","id":5438223,"node_id":"MDQ6VXNlcjU0MzgyMjM=","avatar_url":"https://avatars1.githubusercontent.com/u/5438223?v=4","gravatar_id":"","url":"https://api.github.com/users/kyuksel","html_url":"https://github.com/kyuksel","followers_url":"https://api.github.com/users/kyuksel/followers","following_url":"https://api.github.com/users/kyuksel/following{/other_user}","gists_url":"https://api.github.com/users/kyuksel/gists{/gist_id}","starred_url":"https://api.github.com/users/kyuksel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyuksel/subscriptions","organizations_url":"https://api.github.com/users/kyuksel/orgs","repos_url":"https://api.github.com/users/kyuksel/repos","events_url":"https://api.github.com/users/kyuksel/events{/privacy}","received_events_url":"https://api.github.com/users/kyuksel/received_events","type":"User","site_admin":false},"labels":[{"id":816909070,"node_id":"MDU6TGFiZWw4MTY5MDkwNzA=","url":"https://api.github.com/repos/DataBiosphere/leonardo/labels/%E2%9A%A1%20HEADS%20UP%20%E2%9A%A1","name":"⚡ HEADS UP ⚡","color":"e27526","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-05T19:17:41Z","updated_at":"2018-10-22T15:00:29Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"This is from `gce-leonardo-perf201`, potentially due to the ASHG load testing having created a lot of clusters and/or the free credit projects expiring.\n\n```\n[WARN] [18:59:21.126] [leonardo-akka.actor.default-dispatcher-6018] o.b.d.w.l.m.ZombieClusterMonitor - Unable to check status of project fccredits-thorium-puce-7881 for zombie cluster detection\ncom.google.api.client.googleapis.json.GoogleJsonResponseException: 429 Too Many Requests\n{\n  \"code\" : 429,\n  \"errors\" : [ {\n    \"domain\" : \"global\",\n    \"message\" : \"Quota exceeded for quota metric 'cloudbilling.googleapis.com/standard_quota' and limit 'StandardQuotaPerMinutePerProject' of service 'cloudbilling.googleapis.com' for consumer 'project_number:635957978953'.\",\n    \"reason\" : \"rateLimitExceeded\"\n  } ],\n  \"message\" : \"Quota exceeded for quota metric 'cloudbilling.googleapis.com/standard_quota' and limit 'StandardQuotaPerMinutePerProject' of service 'cloudbilling.googleapis.com' for consumer 'project_number:635957978953'.\",\n  \"status\" : \"RESOURCE_EXHAUSTED\"\n}\n        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)\n        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)\n        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)\n        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)\n        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)\n        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)\n        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)\n        at org.broadinstitute.dsde.workbench.google.GoogleUtilities.$anonfun$executeGoogleCall$1(GoogleUtilities.scala:70)\n        at scala.util.Try$.apply(Try.scala:209)\n        at org.broadinstitute.dsde.workbench.google.GoogleUtilities.executeGoogleCall(GoogleUtilities.scala:70)\n        at org.broadinstitute.dsde.workbench.google.GoogleUtilities.executeGoogleCall$(GoogleUtilities.scala:67)\n        at org.broadinstitute.dsde.workbench.google.AbstractHttpGoogleDAO.executeGoogleCall(AbstractHttpGoogleDAO.scala:14)\n        at org.broadinstitute.dsde.workbench.google.GoogleUtilities.executeGoogleRequest(GoogleUtilities.scala:51)\n        at org.broadinstitute.dsde.workbench.google.GoogleUtilities.executeGoogleRequest$(GoogleUtilities.scala:50)\n        at org.broadinstitute.dsde.workbench.google.AbstractHttpGoogleDAO.executeGoogleRequest(AbstractHttpGoogleDAO.scala:14)\n        at org.broadinstitute.dsde.workbench.google.HttpGoogleProjectDAO.$anonfun$isBillingActive$1(HttpGoogleProjectDAO.scala:65)\n        at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)\n        at akka.dispatch.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3641)\n        at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)\n        at akka.dispatch.BatchingExecutor$BlockableBatch.blockOn(BatchingExecutor.scala:106)\n        at scala.concurrent.package$.blocking(package.scala:142)\n        at org.broadinstitute.dsde.workbench.google.GoogleUtilities.$anonfun$retryWithRecoverWhen500orGoogleError$3(GoogleUtilities.scala:46)\n        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:654)\n        at scala.util.Success.$anonfun$map$1(Try.scala:251)\n        at scala.util.Success.map(Try.scala:209)\n        at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)\n        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)\n        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)\n        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)\n        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)\n        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)\n        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)\n        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)\n        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)\n        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)\n        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)\n        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\n        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\n        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\n```"},"comment":{"url":"https://api.github.com/repos/DataBiosphere/leonardo/issues/comments/431860865","html_url":"https://github.com/DataBiosphere/leonardo/issues/619#issuecomment-431860865","issue_url":"https://api.github.com/repos/DataBiosphere/leonardo/issues/619","id":431860865,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTg2MDg2NQ==","user":{"login":"kyuksel","id":5438223,"node_id":"MDQ6VXNlcjU0MzgyMjM=","avatar_url":"https://avatars1.githubusercontent.com/u/5438223?v=4","gravatar_id":"","url":"https://api.github.com/users/kyuksel","html_url":"https://github.com/kyuksel","followers_url":"https://api.github.com/users/kyuksel/followers","following_url":"https://api.github.com/users/kyuksel/following{/other_user}","gists_url":"https://api.github.com/users/kyuksel/gists{/gist_id}","starred_url":"https://api.github.com/users/kyuksel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyuksel/subscriptions","organizations_url":"https://api.github.com/users/kyuksel/orgs","repos_url":"https://api.github.com/users/kyuksel/repos","events_url":"https://api.github.com/users/kyuksel/events{/privacy}","received_events_url":"https://api.github.com/users/kyuksel/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T15:00:29Z","updated_at":"2018-10-22T15:00:29Z","author_association":"CONTRIBUTOR","body":"Research what the quota is and see if retrying could help."}}
{"actor":{"display_login":"kyuksel"}}