{"action":"created","issue":{"url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3258","repository_url":"https://api.github.com/repos/spinnaker/spinnaker","labels_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3258/labels{/name}","comments_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3258/comments","events_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3258/events","html_url":"https://github.com/spinnaker/spinnaker/issues/3258","id":354721044,"node_id":"MDU6SXNzdWUzNTQ3MjEwNDQ=","number":3258,"title":"GCE: Clouddriver caching failure for TCP Network Load Balancer","user":{"login":"sweetib","id":6327761,"node_id":"MDQ6VXNlcjYzMjc3NjE=","avatar_url":"https://avatars3.githubusercontent.com/u/6327761?v=4","gravatar_id":"","url":"https://api.github.com/users/sweetib","html_url":"https://github.com/sweetib","followers_url":"https://api.github.com/users/sweetib/followers","following_url":"https://api.github.com/users/sweetib/following{/other_user}","gists_url":"https://api.github.com/users/sweetib/gists{/gist_id}","starred_url":"https://api.github.com/users/sweetib/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sweetib/subscriptions","organizations_url":"https://api.github.com/users/sweetib/orgs","repos_url":"https://api.github.com/users/sweetib/repos","events_url":"https://api.github.com/users/sweetib/events{/privacy}","received_events_url":"https://api.github.com/users/sweetib/received_events","type":"User","site_admin":false},"labels":[{"id":111166987,"node_id":"MDU6TGFiZWwxMTExNjY5ODc=","url":"https://api.github.com/repos/spinnaker/spinnaker/labels/bug","name":"bug","color":"fc4400","default":true},{"id":288417181,"node_id":"MDU6TGFiZWwyODg0MTcxODE=","url":"https://api.github.com/repos/spinnaker/spinnaker/labels/component/clouddriver","name":"component/clouddriver","color":"c7def8","default":false},{"id":288417117,"node_id":"MDU6TGFiZWwyODg0MTcxMTc=","url":"https://api.github.com/repos/spinnaker/spinnaker/labels/provider/gce","name":"provider/gce","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-28T13:02:03Z","updated_at":"2018-08-28T16:43:21Z","closed_at":null,"author_association":"NONE","body":"### Issue Summary:\r\nCurrently we have started getting frequent errors while fetching details about existing network load balancers in project.\r\n\r\n### Cloud Provider(s):\r\nGCE\r\n\r\n### Environment:\r\nUbuntu 16.04 xenial\r\n\r\n### Feature Area: \r\nDeck, Clouddriver\r\n\r\n### Description:\r\nWe are using spinnaker 1.8.2 . We added new gcp project under spinnaker. We had several lbs already created from gcp console that gets populated for the application related to this project. Some how the UI is flaky and keeps showing and not showing the load balancer related details\r\n \r\nDue to this we are not able to perform deployments behind this load balancer \r\n\r\n**Error Details**\r\n\r\nHere there are two lbs\r\n- abc-external-beta-x\r\n- abc-external-beta\r\nwith 3 forwarding rule each for port 443,5066,6066\r\n\r\nclouddriver logs\r\n```\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: 2018-08-28 12:34:03.535  WARN 1248 --- [utionAction-536] .a.GoogleNetworkLoadBalancerCachingAgent : Failed to read a component of subject abc-external-beta-x-443-fw. The platform error message was:\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]:  The service is currently unavailable..\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: Reporting it as 'Failed' to the caching agent.\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: 2018-08-28 12:34:03.535  WARN 1248 --- [utionAction-536] .a.GoogleNetworkLoadBalancerCachingAgent : Failed to read a component of subject abc-external-beta-x-5066-fw. The platform error message was:\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]:  The service is currently unavailable..\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: Reporting it as 'Failed' to the caching agent.\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: 2018-08-28 12:34:03.535  WARN 1248 --- [utionAction-536] .a.GoogleNetworkLoadBalancerCachingAgent : Failed to read a component of subject abc-external-beta-x-6066-fw. The platform error message was:\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]:  The service is currently unavailable..\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: Reporting it as 'Failed' to the caching agent.\r\n\r\n\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: 2018-08-28 12:34:03.536  WARN 1248 --- [utionAction-536] .a.GoogleNetworkLoadBalancerCachingAgent : Failed to read a component of subject abc-external-beta-443-fw. The platform error message was:\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]:  The service is currently unavailable..\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: Reporting it as 'Failed' to the caching agent.\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: 2018-08-28 12:34:03.536  WARN 1248 --- [utionAction-536] .a.GoogleNetworkLoadBalancerCachingAgent : Failed to read a component of subject abc-external-beta-5066-fw. The platform error message was:\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]:  The service is currently unavailable..\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: Reporting it as 'Failed' to the caching agent.\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: 2018-08-28 12:34:03.536  WARN 1248 --- [utionAction-536] .a.GoogleNetworkLoadBalancerCachingAgent : Failed to read a component of subject abc-external-beta-6066-fw. The platform error message was:\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]:  The service is currently unavailable..\r\nAug 28 12:34:03 clouddriver-halyard clouddriver[1248]: Reporting it as 'Failed' to the caching agent.\r\n```\r\n\r\nAs we are facing this issue in production this is blocking our deployments. Awaiting fix or any solution for this issue"},"comment":{"url":"https://api.github.com/repos/spinnaker/spinnaker/issues/comments/416657408","html_url":"https://github.com/spinnaker/spinnaker/issues/3258#issuecomment-416657408","issue_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3258","id":416657408,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjY1NzQwOA==","user":{"login":"ezimanyi","id":1391982,"node_id":"MDQ6VXNlcjEzOTE5ODI=","avatar_url":"https://avatars3.githubusercontent.com/u/1391982?v=4","gravatar_id":"","url":"https://api.github.com/users/ezimanyi","html_url":"https://github.com/ezimanyi","followers_url":"https://api.github.com/users/ezimanyi/followers","following_url":"https://api.github.com/users/ezimanyi/following{/other_user}","gists_url":"https://api.github.com/users/ezimanyi/gists{/gist_id}","starred_url":"https://api.github.com/users/ezimanyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ezimanyi/subscriptions","organizations_url":"https://api.github.com/users/ezimanyi/orgs","repos_url":"https://api.github.com/users/ezimanyi/repos","events_url":"https://api.github.com/users/ezimanyi/events{/privacy}","received_events_url":"https://api.github.com/users/ezimanyi/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T16:43:21Z","updated_at":"2018-08-28T16:43:21Z","author_association":"MEMBER","body":"The caching changes in https://github.com/spinnaker/clouddriver/pull/2811 improved logging of cases when we're not properly caching an HTTPS load balancer.  (It didn't affect network load balancers, as those already were properly handling and logging failures.)\r\n\r\nWhat it looks like is happening here is that the caching agent is failing when communicating with GCP to request information about the load balancers.  In particular, it appears that GCP is returning \"The service is currently unavailable.\" as a response to Spinnaker's requests.\r\n\r\nWe'll have to figure out why the requests are returning that response.  One thought, is it's possibly you may be exceeding operation quota for the GCP project.  Can you check here:\r\nhttps://console.cloud.google.com/apis/api/compute.googleapis.com/quotas?project=your-project\r\n(replacing your-project in the URL with the appropriate project) to see if any of the operation types are approaching or exceeding their quota?\r\n\r\nThat's just my first thought---if that's not the issue, we can try to dig more and figure out what is going on."}}
{"actor":{"display_login":"ezimanyi"}}