{"action":"created","comment":{"url":"https://api.github.com/repos/spinnaker/fiat/pulls/comments/80499264","id":80499264,"diff_hunk":"@@ -62,57 +76,78 @@ public void sync() {\n   }\n \n   public long syncAndReturn() {\n-    updateServiceAccounts();\n-    return updateUserPermissions();\n-  }\n-\n-  private void updateServiceAccounts() {\n-    AtomicInteger updateCount = new AtomicInteger(0);\n-\n     FixedBackOff backoff = new FixedBackOff();\n     backoff.setInterval(retryIntervalMs);\n     BackOffExecution backOffExec = backoff.start();\n \n+    if (!isServerHealthy()) {\n+      log.warn(\"Server is currently UNHEALTHY. User permission role synchronization and \" +\n+                   \"resolution may not complete until this server becomes healthy again.\");","path":"fiat-roles/src/main/java/com/netflix/spinnaker/fiat/roles/UserRolesSyncer.java","position":62,"original_position":62,"commit_id":"af438fb2e77139779ba6782beb657a8284395740","original_commit_id":"af438fb2e77139779ba6782beb657a8284395740","user":{"login":"ttomsu","id":13141550,"avatar_url":"https://avatars.githubusercontent.com/u/13141550?v=3","gravatar_id":"","url":"https://api.github.com/users/ttomsu","html_url":"https://github.com/ttomsu","followers_url":"https://api.github.com/users/ttomsu/followers","following_url":"https://api.github.com/users/ttomsu/following{/other_user}","gists_url":"https://api.github.com/users/ttomsu/gists{/gist_id}","starred_url":"https://api.github.com/users/ttomsu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ttomsu/subscriptions","organizations_url":"https://api.github.com/users/ttomsu/orgs","repos_url":"https://api.github.com/users/ttomsu/repos","events_url":"https://api.github.com/users/ttomsu/events{/privacy}","received_events_url":"https://api.github.com/users/ttomsu/received_events","type":"User","site_admin":false},"body":"It won't mean anything for requests - but when used behind a load balancer, traffic will halt once the LB detects the instance has gone unhealthy. That has been the case ever since I introduced the `ResourceProviderHealthIndicator`, but now it's surfaced more in the logs for debugging.","created_at":"2016-09-26T15:18:06Z","updated_at":"2016-09-26T15:18:07Z","html_url":"https://github.com/spinnaker/fiat/pull/69#discussion_r80499264","pull_request_url":"https://api.github.com/repos/spinnaker/fiat/pulls/69","_links":{"self":{"href":"https://api.github.com/repos/spinnaker/fiat/pulls/comments/80499264"},"html":{"href":"https://github.com/spinnaker/fiat/pull/69#discussion_r80499264"},"pull_request":{"href":"https://api.github.com/repos/spinnaker/fiat/pulls/69"}}},"pull_request":{"url":"https://api.github.com/repos/spinnaker/fiat/pulls/69","id":86657215,"html_url":"https://github.com/spinnaker/fiat/pull/69","diff_url":"https://github.com/spinnaker/fiat/pull/69.diff","patch_url":"https://github.com/spinnaker/fiat/pull/69.patch","issue_url":"https://api.github.com/repos/spinnaker/fiat/issues/69","number":69,"state":"open","locked":false,"title":"Improves startup behavior when dependent services are down.","user":{"login":"ttomsu","id":13141550,"avatar_url":"https://avatars.githubusercontent.com/u/13141550?v=3","gravatar_id":"","url":"https://api.github.com/users/ttomsu","html_url":"https://github.com/ttomsu","followers_url":"https://api.github.com/users/ttomsu/followers","following_url":"https://api.github.com/users/ttomsu/following{/other_user}","gists_url":"https://api.github.com/users/ttomsu/gists{/gist_id}","starred_url":"https://api.github.com/users/ttomsu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ttomsu/subscriptions","organizations_url":"https://api.github.com/users/ttomsu/orgs","repos_url":"https://api.github.com/users/ttomsu/repos","events_url":"https://api.github.com/users/ttomsu/events{/privacy}","received_events_url":"https://api.github.com/users/ttomsu/received_events","type":"User","site_admin":false},"body":"Fixes https://github.com/spinnaker/fiat/issues/46\r\n\r\nFiat is heavily dependent on Front50 and Clouddriver to get bootstrapped data. Previous efforts ensured Front50 was up in order to resolve service accounts, and included a HealthIndicator to the load balancer to ensure traffic wasn't sent until Fiat was able to connect to all resource providers.\r\n\r\nHealth checks aren't really helpful enough for single instance and/or load-balancer-less development, especially if Front50 was up, but Clouddriver down, because there was minimal logging indicating a problem.\r\n\r\nThe solution is to multi-fold:\r\n* Merge and simplify the UserRoleSyncer's `updateServiceAccounts` with its `updateUserPermissions`.\r\n* Changes the `PermissionResolver` interface to throw exceptions instead of returning `Optional`, because it turns out having that exception adds a lot of code flow control benefits.\r\n* Use the `ResourceProviderHealthIndicator` to notify via log statements when the server goes in and out of the `Status.UP` state.\r\n\r\n\r\nThe log output is now much more helpful. For example, at server startup:\r\n\r\n```\r\n2016-09-25 13:25:38.473  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : Server is currently UNHEALTHY. User permission role synchronization and resolution may not complete until this server becomes healthy again.\r\n2016-09-25 13:25:38.505  INFO 6479 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 7003 (http)\r\n2016-09-25 13:25:38.507  INFO 6479 --- [           main] com.netflix.spinnaker.fiat.Main          : Started Main in 4.972 seconds (JVM running for 5.508)\r\n2016-09-25 13:25:39.105  INFO 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : Synced anonymous user role.\r\n2016-09-25 13:25:39.876  INFO 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : Synced 1 non-anonymous user roles.\r\n2016-09-25 13:25:39.878  INFO 6479 --- [pool-1-thread-1] n.s.f.c.ResourceProvidersHealthIndicator : Server is now HEALTHY. Hooray!\r\n```\r\n\r\nIf (and when) a dependent service goes down (in this example, Clouddriver), the following gets logged:\r\n```\r\n2016-09-25 13:35:40.057  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is UP. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n2016-09-25 13:35:45.203  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is UP. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n2016-09-25 13:35:50.359  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is UP. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n2016-09-25 13:35:55.512  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is UP. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n2016-09-25 13:36:00.736  WARN 6479 --- [pool-1-thread-1] n.s.f.c.ResourceProvidersHealthIndicator : Server is now UNHEALTHY\r\n2016-09-25 13:36:00.736  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is DOWN. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n2016-09-25 13:36:05.920  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is DOWN. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n...\r\n```\r\nand will continue every (configurable) 5s until the resource becomes healthy again:\r\n```\r\n...\r\n2016-09-25 13:37:59.505  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is DOWN. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n2016-09-25 13:38:04.740  WARN 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : User permission sync failed. Server status is DOWN. Trying again in 5000 ms. Cause:com.netflix.spinnaker.fiat.providers.ProviderException: retrofit.RetrofitError: Failed to connect to localhost/0:0:0:0:0:0:0:1:7002\r\n2016-09-25 13:38:10.266  INFO 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : Synced anonymous user role.\r\n2016-09-25 13:38:10.721  INFO 6479 --- [pool-1-thread-1] c.n.s.fiat.roles.UserRolesSyncer         : Synced 1 non-anonymous user roles.\r\n2016-09-25 13:38:10.721  INFO 6479 --- [pool-1-thread-1] n.s.f.c.ResourceProvidersHealthIndicator : Server is now HEALTHY. Hooray!\r\n```\r\n\r\n@jtk54 @duftler @cfieber @ajordens PTAL\r\n\r\n","created_at":"2016-09-25T17:54:09Z","updated_at":"2016-09-26T15:18:07Z","closed_at":null,"merged_at":null,"merge_commit_sha":"333c2c7b0e0edabef4d06128a926a773f6d26eef","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/spinnaker/fiat/pulls/69/commits","review_comments_url":"https://api.github.com/repos/spinnaker/fiat/pulls/69/comments","review_comment_url":"https://api.github.com/repos/spinnaker/fiat/pulls/comments{/number}","comments_url":"https://api.github.com/repos/spinnaker/fiat/issues/69/comments","statuses_url":"https://api.github.com/repos/spinnaker/fiat/statuses/af438fb2e77139779ba6782beb657a8284395740","head":{"label":"spinnaker:google-better-startup","ref":"google-better-startup","sha":"af438fb2e77139779ba6782beb657a8284395740","user":{"login":"spinnaker","id":7634182,"avatar_url":"https://avatars.githubusercontent.com/u/7634182?v=3","gravatar_id":"","url":"https://api.github.com/users/spinnaker","html_url":"https://github.com/spinnaker","followers_url":"https://api.github.com/users/spinnaker/followers","following_url":"https://api.github.com/users/spinnaker/following{/other_user}","gists_url":"https://api.github.com/users/spinnaker/gists{/gist_id}","starred_url":"https://api.github.com/users/spinnaker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spinnaker/subscriptions","organizations_url":"https://api.github.com/users/spinnaker/orgs","repos_url":"https://api.github.com/users/spinnaker/repos","events_url":"https://api.github.com/users/spinnaker/events{/privacy}","received_events_url":"https://api.github.com/users/spinnaker/received_events","type":"Organization","site_admin":false},"repo":{"id":61139592,"name":"fiat","full_name":"spinnaker/fiat","owner":{"login":"spinnaker","id":7634182,"avatar_url":"https://avatars.githubusercontent.com/u/7634182?v=3","gravatar_id":"","url":"https://api.github.com/users/spinnaker","html_url":"https://github.com/spinnaker","followers_url":"https://api.github.com/users/spinnaker/followers","following_url":"https://api.github.com/users/spinnaker/following{/other_user}","gists_url":"https://api.github.com/users/spinnaker/gists{/gist_id}","starred_url":"https://api.github.com/users/spinnaker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spinnaker/subscriptions","organizations_url":"https://api.github.com/users/spinnaker/orgs","repos_url":"https://api.github.com/users/spinnaker/repos","events_url":"https://api.github.com/users/spinnaker/events{/privacy}","received_events_url":"https://api.github.com/users/spinnaker/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/spinnaker/fiat","description":"Spinnaker auth service","fork":false,"url":"https://api.github.com/repos/spinnaker/fiat","forks_url":"https://api.github.com/repos/spinnaker/fiat/forks","keys_url":"https://api.github.com/repos/spinnaker/fiat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/spinnaker/fiat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/spinnaker/fiat/teams","hooks_url":"https://api.github.com/repos/spinnaker/fiat/hooks","issue_events_url":"https://api.github.com/repos/spinnaker/fiat/issues/events{/number}","events_url":"https://api.github.com/repos/spinnaker/fiat/events","assignees_url":"https://api.github.com/repos/spinnaker/fiat/assignees{/user}","branches_url":"https://api.github.com/repos/spinnaker/fiat/branches{/branch}","tags_url":"https://api.github.com/repos/spinnaker/fiat/tags","blobs_url":"https://api.github.com/repos/spinnaker/fiat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/spinnaker/fiat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/spinnaker/fiat/git/refs{/sha}","trees_url":"https://api.github.com/repos/spinnaker/fiat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/spinnaker/fiat/statuses/{sha}","languages_url":"https://api.github.com/repos/spinnaker/fiat/languages","stargazers_url":"https://api.github.com/repos/spinnaker/fiat/stargazers","contributors_url":"https://api.github.com/repos/spinnaker/fiat/contributors","subscribers_url":"https://api.github.com/repos/spinnaker/fiat/subscribers","subscription_url":"https://api.github.com/repos/spinnaker/fiat/subscription","commits_url":"https://api.github.com/repos/spinnaker/fiat/commits{/sha}","git_commits_url":"https://api.github.com/repos/spinnaker/fiat/git/commits{/sha}","comments_url":"https://api.github.com/repos/spinnaker/fiat/comments{/number}","issue_comment_url":"https://api.github.com/repos/spinnaker/fiat/issues/comments{/number}","contents_url":"https://api.github.com/repos/spinnaker/fiat/contents/{+path}","compare_url":"https://api.github.com/repos/spinnaker/fiat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/spinnaker/fiat/merges","archive_url":"https://api.github.com/repos/spinnaker/fiat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/spinnaker/fiat/downloads","issues_url":"https://api.github.com/repos/spinnaker/fiat/issues{/number}","pulls_url":"https://api.github.com/repos/spinnaker/fiat/pulls{/number}","milestones_url":"https://api.github.com/repos/spinnaker/fiat/milestones{/number}","notifications_url":"https://api.github.com/repos/spinnaker/fiat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/spinnaker/fiat/labels{/name}","releases_url":"https://api.github.com/repos/spinnaker/fiat/releases{/id}","deployments_url":"https://api.github.com/repos/spinnaker/fiat/deployments","created_at":"2016-06-14T16:44:56Z","updated_at":"2016-09-19T20:28:45Z","pushed_at":"2016-09-25T17:54:09Z","git_url":"git://github.com/spinnaker/fiat.git","ssh_url":"git@github.com:spinnaker/fiat.git","clone_url":"https://github.com/spinnaker/fiat.git","svn_url":"https://github.com/spinnaker/fiat","homepage":null,"size":291,"stargazers_count":9,"watchers_count":9,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":9,"mirror_url":null,"open_issues_count":14,"forks":9,"open_issues":14,"watchers":9,"default_branch":"master"}},"base":{"label":"spinnaker:master","ref":"master","sha":"934e6ccb96ef4794b22784fc13fa30cd4257bd47","user":{"login":"spinnaker","id":7634182,"avatar_url":"https://avatars.githubusercontent.com/u/7634182?v=3","gravatar_id":"","url":"https://api.github.com/users/spinnaker","html_url":"https://github.com/spinnaker","followers_url":"https://api.github.com/users/spinnaker/followers","following_url":"https://api.github.com/users/spinnaker/following{/other_user}","gists_url":"https://api.github.com/users/spinnaker/gists{/gist_id}","starred_url":"https://api.github.com/users/spinnaker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spinnaker/subscriptions","organizations_url":"https://api.github.com/users/spinnaker/orgs","repos_url":"https://api.github.com/users/spinnaker/repos","events_url":"https://api.github.com/users/spinnaker/events{/privacy}","received_events_url":"https://api.github.com/users/spinnaker/received_events","type":"Organization","site_admin":false},"repo":{"id":61139592,"name":"fiat","full_name":"spinnaker/fiat","owner":{"login":"spinnaker","id":7634182,"avatar_url":"https://avatars.githubusercontent.com/u/7634182?v=3","gravatar_id":"","url":"https://api.github.com/users/spinnaker","html_url":"https://github.com/spinnaker","followers_url":"https://api.github.com/users/spinnaker/followers","following_url":"https://api.github.com/users/spinnaker/following{/other_user}","gists_url":"https://api.github.com/users/spinnaker/gists{/gist_id}","starred_url":"https://api.github.com/users/spinnaker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spinnaker/subscriptions","organizations_url":"https://api.github.com/users/spinnaker/orgs","repos_url":"https://api.github.com/users/spinnaker/repos","events_url":"https://api.github.com/users/spinnaker/events{/privacy}","received_events_url":"https://api.github.com/users/spinnaker/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/spinnaker/fiat","description":"Spinnaker auth service","fork":false,"url":"https://api.github.com/repos/spinnaker/fiat","forks_url":"https://api.github.com/repos/spinnaker/fiat/forks","keys_url":"https://api.github.com/repos/spinnaker/fiat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/spinnaker/fiat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/spinnaker/fiat/teams","hooks_url":"https://api.github.com/repos/spinnaker/fiat/hooks","issue_events_url":"https://api.github.com/repos/spinnaker/fiat/issues/events{/number}","events_url":"https://api.github.com/repos/spinnaker/fiat/events","assignees_url":"https://api.github.com/repos/spinnaker/fiat/assignees{/user}","branches_url":"https://api.github.com/repos/spinnaker/fiat/branches{/branch}","tags_url":"https://api.github.com/repos/spinnaker/fiat/tags","blobs_url":"https://api.github.com/repos/spinnaker/fiat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/spinnaker/fiat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/spinnaker/fiat/git/refs{/sha}","trees_url":"https://api.github.com/repos/spinnaker/fiat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/spinnaker/fiat/statuses/{sha}","languages_url":"https://api.github.com/repos/spinnaker/fiat/languages","stargazers_url":"https://api.github.com/repos/spinnaker/fiat/stargazers","contributors_url":"https://api.github.com/repos/spinnaker/fiat/contributors","subscribers_url":"https://api.github.com/repos/spinnaker/fiat/subscribers","subscription_url":"https://api.github.com/repos/spinnaker/fiat/subscription","commits_url":"https://api.github.com/repos/spinnaker/fiat/commits{/sha}","git_commits_url":"https://api.github.com/repos/spinnaker/fiat/git/commits{/sha}","comments_url":"https://api.github.com/repos/spinnaker/fiat/comments{/number}","issue_comment_url":"https://api.github.com/repos/spinnaker/fiat/issues/comments{/number}","contents_url":"https://api.github.com/repos/spinnaker/fiat/contents/{+path}","compare_url":"https://api.github.com/repos/spinnaker/fiat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/spinnaker/fiat/merges","archive_url":"https://api.github.com/repos/spinnaker/fiat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/spinnaker/fiat/downloads","issues_url":"https://api.github.com/repos/spinnaker/fiat/issues{/number}","pulls_url":"https://api.github.com/repos/spinnaker/fiat/pulls{/number}","milestones_url":"https://api.github.com/repos/spinnaker/fiat/milestones{/number}","notifications_url":"https://api.github.com/repos/spinnaker/fiat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/spinnaker/fiat/labels{/name}","releases_url":"https://api.github.com/repos/spinnaker/fiat/releases{/id}","deployments_url":"https://api.github.com/repos/spinnaker/fiat/deployments","created_at":"2016-06-14T16:44:56Z","updated_at":"2016-09-19T20:28:45Z","pushed_at":"2016-09-25T17:54:09Z","git_url":"git://github.com/spinnaker/fiat.git","ssh_url":"git@github.com:spinnaker/fiat.git","clone_url":"https://github.com/spinnaker/fiat.git","svn_url":"https://github.com/spinnaker/fiat","homepage":null,"size":291,"stargazers_count":9,"watchers_count":9,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":9,"mirror_url":null,"open_issues_count":14,"forks":9,"open_issues":14,"watchers":9,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/spinnaker/fiat/pulls/69"},"html":{"href":"https://github.com/spinnaker/fiat/pull/69"},"issue":{"href":"https://api.github.com/repos/spinnaker/fiat/issues/69"},"comments":{"href":"https://api.github.com/repos/spinnaker/fiat/issues/69/comments"},"review_comments":{"href":"https://api.github.com/repos/spinnaker/fiat/pulls/69/comments"},"review_comment":{"href":"https://api.github.com/repos/spinnaker/fiat/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/spinnaker/fiat/pulls/69/commits"},"statuses":{"href":"https://api.github.com/repos/spinnaker/fiat/statuses/af438fb2e77139779ba6782beb657a8284395740"}}}}
{"actor":{"display_login":"ttomsu"}}