{"action":"created","issue":{"url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3287","repository_url":"https://api.github.com/repos/spinnaker/spinnaker","labels_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3287/labels{/name}","comments_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3287/comments","events_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3287/events","html_url":"https://github.com/spinnaker/spinnaker/issues/3287","id":356964126,"node_id":"MDU6SXNzdWUzNTY5NjQxMjY=","number":3287,"title":"Clouddriver: Unready - java.lang.OutOfMemoryError","user":{"login":"louisvernon","id":552230,"node_id":"MDQ6VXNlcjU1MjIzMA==","avatar_url":"https://avatars2.githubusercontent.com/u/552230?v=4","gravatar_id":"","url":"https://api.github.com/users/louisvernon","html_url":"https://github.com/louisvernon","followers_url":"https://api.github.com/users/louisvernon/followers","following_url":"https://api.github.com/users/louisvernon/following{/other_user}","gists_url":"https://api.github.com/users/louisvernon/gists{/gist_id}","starred_url":"https://api.github.com/users/louisvernon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/louisvernon/subscriptions","organizations_url":"https://api.github.com/users/louisvernon/orgs","repos_url":"https://api.github.com/users/louisvernon/repos","events_url":"https://api.github.com/users/louisvernon/events{/privacy}","received_events_url":"https://api.github.com/users/louisvernon/received_events","type":"User","site_admin":false},"labels":[{"id":339776236,"node_id":"MDU6TGFiZWwzMzk3NzYyMzY=","url":"https://api.github.com/repos/spinnaker/spinnaker/labels/operations/installation","name":"operations/installation","color":"fbca04","default":false},{"id":916190248,"node_id":"MDU6TGFiZWw5MTYxOTAyNDg=","url":"https://api.github.com/repos/spinnaker/spinnaker/labels/provider/kubernetes-v2","name":"provider/kubernetes-v2","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-09-04T20:29:00Z","updated_at":"2018-10-25T19:32:33Z","closed_at":null,"author_association":"NONE","body":"### Issue Summary:\r\nAfter upgrading from 1.8.4 to 1.9.2 we find that after some time Clouddriver memory usage abruptly starts to climb to ~10GB before Clouddriver falls over and reports Unready in GKE. Clouddriver remains Unready until the pod is terminated and the process is repeated. \r\n\r\n### Cloud Provider(s):\r\n\r\nGKE \r\n\r\n### Environment:\r\n\r\nNode version\t\r\n1.10.4-gke.2 \r\nNode image\t\r\nContainer-Optimized OS (cos) \r\nMachine type\t\r\nn1-standard-8 (8 vCPUs, 30 GB memory)\r\nTotal cores\t\r\n16 vCPUs\r\nTotal memory\t\r\n60.00 GB\r\n.\r\nSpinnaker has been deployed to a cluster configured using Kubernetes v1 provider, while managing several other GKE clusters using Kubernetes v2 provider.\r\n\r\n### Feature Area: \r\n\r\nClouddriver\r\n\r\n### Additional Details:\r\n\r\nWe never knowingly experienced memory issues under previous Spinnaker releases. `free -mh` and historical metrics suggest that we have some spare capacity (~15%) at the point of failure, although we are provisioning larger nodes now.\r\n\r\nResource usage graph:\r\n\r\nCPU fall-off coincides with Clouddriver unreadiness.\r\n![image](https://user-images.githubusercontent.com/552230/45054967-cc371d80-b04b-11e8-9b09-6237b7686fa3.png)\r\n\r\n\r\nLog exerpts:\r\n```\r\nI  \r\n \r\nI  org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:982) ~[spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.5.10.RELEASE.jar:1.5.10.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:110) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) [spring-boot-actuator-1.5.10.RELEASE.jar:1.5.10.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat com.netflix.spinnaker.fiat.shared.FiatAuthenticationFilter.doFilter(FiatAuthenticationFilter.java:46) [fiat-api-0.49.6.jar:0.49.6]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) [spring-security-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]\r\n\tat org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) [spring-boot-actuator-1.5.10.RELEASE.jar:1.5.10.RELEASE]\r\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat javax.servlet.FilterChain$doFilter.call(Unknown Source) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat com.netflix.spinnaker.filters.AuthenticatedRequestFilter.doFilter(AuthenticatedRequestFilter.groovy:135) [kork-web-2.0.0.jar:2.0.0]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]\r\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.27.jar:8.5.27]\r\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]\r\nCaused by: java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:18:49.882 ERROR 1 --- [.0-7002-exec-12] c.n.s.k.w.e.GenericExceptionHandlers     : Internal Server Error\r\n \r\nE  Exception in thread \"Exec Stream Pumper\" java.lang.OutOfMemoryError: Java heap space\r\n \r\nI  2018-09-04 18:18:17.055  INFO 1 --- [utionAction-597] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:gke_dl-platform_us-central1-b_platform-labv2/KubernetesNamespaceCachingAgent[1/1] completed in 29.51s\r\n \r\nI  2018-09-04 18:18:14.748  INFO 1 --- [utionAction-605] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesConfigMapCachingAgent[1/1] completed in 132.956s\r\n \r\nI  2018-09-04 18:18:14.746  INFO 1 --- [utionAction-597] s.c.k.v.c.a.KubernetesCacheDataConverter : gke_dl-platform_us-central1-b_platform-labv2/KubernetesNamespaceCachingAgent[1/1]: grouping applications has 10 entries and 11 relationships\r\n \r\nI  2018-09-04 18:18:14.746  INFO 1 --- [utionAction-597] s.c.k.v.c.a.KubernetesCacheDataConverter : gke_dl-platform_us-central1-b_platform-labv2/KubernetesNamespaceCachingAgent[1/1]: grouping namespace has 11 entries and 11 relationships\r\n \r\nI  2018-09-04 18:18:14.742  INFO 1 --- [utionAction-605] .k.v.p.a.KubernetesConfigMapCachingAgent : Caching 9 configmaps in spinnaker/KubernetesConfigMapCachingAgent[1/1]\r\n \r\nI  2018-09-04 18:18:14.740  INFO 1 --- [utionAction-606] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesDeploymentCachingAgent[1/1] completed in 481.174s\r\n \r\nI  2018-09-04 18:18:14.740  INFO 1 --- [utionAction-517] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesSecurityGroupCachingAgent[1/1] completed in 28.275s\r\n \r\nI  2018-09-04 18:18:12.714  INFO 1 --- [utionAction-585] c.k.v.p.a.KubernetesInstanceCachingAgent : Describing items in spinnaker/KubernetesInstanceCachingAgent[1/1]\r\n \r\nI  2018-09-04 18:18:12.710  INFO 1 --- [utionAction-605] .k.v.p.a.KubernetesConfigMapCachingAgent : Describing items in spinnaker/KubernetesConfigMapCachingAgent[1/1]\r\n \r\nI  2018-09-04 18:18:11.652  INFO 1 --- [utionAction-606] k.v.p.a.KubernetesDeploymentCachingAgent : Caching 9 deployments in spinnaker/KubernetesDeploymentCachingAgent[1/1]\r\n \r\nI  2018-09-04 18:18:11.651  INFO 1 --- [utionAction-517] .p.a.KubernetesSecurityGroupCachingAgent : Caching 0 security groups in spinnaker/KubernetesSecurityGroupCachingAgent[1/1]\r\n \r\nI  2018-09-04 18:18:11.648  INFO 1 --- [utionAction-517] .p.a.KubernetesSecurityGroupCachingAgent : Describing items in spinnaker/KubernetesSecurityGroupCachingAgent[1/1]\r\n \r\nI  2018-09-04 18:18:11.647  INFO 1 --- [utionAction-607] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesSecretCachingAgent[1/1] completed in 229.913s\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:18:07.851  WARN 1 --- [utionAction-609] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.docker.registry.provider.DockerRegistryProvider:spinnaker-usgcr-account/DockerRegistryImageCachingAgent[1/1] completed with one or more failures\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:18:07.841  WARN 1 --- [utionAction-593] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesControllersCachingAgent[1/1] completed with one or more failures\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:18:07.841  WARN 1 --- [utionAction-603] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:gke_dl-platform_us-central1-b_platform-labv2/KubernetesCoreCachingAgent[1/1] completed with one or more failures\r\n \r\nI  2018-09-04 18:18:07.841  INFO 1 --- [utionAction-607] s.c.k.v.p.a.KubernetesSecretCachingAgent : Caching 0 secrets in spinnaker/KubernetesSecretCachingAgent[1/1]\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:17:48.155  WARN 1 --- [utionAction-602] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesLoadBalancerCachingAgent[1/1] completed with one or more failures\r\n \r\nI  \r\n \r\nI  java.lang.NullPointerException: null\r\n \r\nI  \r\n \r\nI  2018-09-04 18:17:47.323  WARN 1 --- [utionAction-604] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:gke_dl-platform_us-central1-b_platform-labv2/KubernetesUnregisteredCustomResourceCachingAgent[1/1] completed with one or more failures\r\n \r\nI  2018-09-04 18:17:46.464  INFO 1 --- [utionAction-597] c.n.s.c.k.v.c.a.KubernetesV2CachingAgent : gke_dl-platform_us-central1-b_platform-labv2/KubernetesNamespaceCachingAgent[1/1] is starting\r\n \r\nI  2018-09-04 18:17:46.463  INFO 1 --- [utionAction-607] s.c.k.v.p.a.KubernetesSecretCachingAgent : Describing items in spinnaker/KubernetesSecretCachingAgent[1/1]\r\n \r\nI  \r\n \r\nI  java.lang.NullPointerException: null\r\n\tat com.netflix.spinnaker.clouddriver.kubernetes.v1.security.KubernetesV1Credentials.getDeclaredNamespaces(KubernetesV1Credentials.java:140) ~[clouddriver-kubernetes-2.67.0-SNAPSHOT.jar:2.67.0-SNAPSHOT]\r\n\tat com.netflix.spinnaker.clouddriver.kubernetes.caching.KubernetesCachingAgent.reloadNamespaces(KubernetesCachingAgent.java:64) [clouddriver-kubernetes-2.67.0-SNAPSHOT.jar:2.67.0-SNAPSHOT]\r\n\tat sun.reflect.GeneratedMethodAccessor1071.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]\r\n\tat java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]\r\n\tat org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210) [groovy-all-2.4.13.jar:2.4.13]\r\n\tat org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59) [groovy-all-2.4.13.jar:2.4.13]\r\n\tat org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52) [groovy-all-2.4.13.jar:2.4.13]\r\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154) [groovy-all-2.4.13.jar:2.4.13]\r\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:158) [groovy-all-2.4.13.jar:2.4.13]\r\n\tat com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.agent.KubernetesSecretCachingAgent.loadData(KubernetesSecretCachingAgent.groovy:56) [clouddriver-kubernetes-2.67.0-SNAPSHOT.jar:2.67.0-SNAPSHOT]\r\n\tat com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgentWithoutStore(CachingAgent.java:80) [cats-core-2.67.0-SNAPSHOT.jar:2.67.0-SNAPSHOT]\r\n\tat com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgent(CachingAgent.java:73) [cats-core-2.67.0-SNAPSHOT.jar:2.67.0-SNAPSHOT]\r\n\tat com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentExecutionAction.execute(ClusteredAgentScheduler.java:299) [cats-redis-2.67.0-SNAPSHOT.jar:2.67.0-SNAPSHOT]\r\n\tat com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentJob.run(ClusteredAgentScheduler.java:273) [cats-redis-2.67.0-SNAPSHOT.jar:2.67.0-SNAPSHOT]\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]\r\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]\r\n \r\nI  \r\n \r\nI  2018-09-04 18:17:37.833  WARN 1 --- [utionAction-607] c.n.s.c.k.v.s.KubernetesV1Credentials    : Could not determine kubernetes namespaces. Will try again later.\r\n \r\nE  Exception in thread \"ContainerBackgroundProcessor[StandardEngine[Tomcat]]\" java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:17:18.508 ERROR 1 --- [Engine[Tomcat]]] org.apache.catalina.core.ContainerBase   : Unexpected death of background thread [ContainerBackgroundProcessor[StandardEngine[Tomcat]]]\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:17:18.508 ERROR 1 --- [igurationSource] c.n.config.AbstractPollingScheduler      : Error getting result from polling source\r\n \r\nE  Exception in thread \"Exec Default Executor\" java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nE  Exception in thread \"Exec Stream Pumper\" java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:16:42.597 ERROR 1 --- [igurationSource] c.n.config.AbstractPollingScheduler      : Error getting result from polling source\r\n \r\nE  Exception in thread \"Exec Stream Pumper\" java.lang.OutOfMemoryError: Java heap space\r\n \r\nE  Exception in thread \"Exec Stream Pumper\" java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  2018-09-04 18:16:01.792  INFO 1 --- [utionAction-605] .k.v.p.a.KubernetesConfigMapCachingAgent : Loading config maps in spinnaker/KubernetesConfigMapCachingAgent[1/1]\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:15:52.187  WARN 1 --- [utionAction-598] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesServiceAccountCachingAgent[1/1] completed with one or more failures\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:15:50.724  WARN 1 --- [utionAction-517] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.KubernetesV1Provider:spinnaker/KubernetesSecurityGroupCachingAgent[1/1] completed with one or more failures\r\n \r\nI  \r\n \r\nI  java.lang.NullPointerException: null\r\n \r\nI  \r\n \r\nI  2018-09-04 18:15:48.454  WARN 1 --- [utionAction-589] c.n.s.c.cache.LoggingInstrumentation     : kubernetes:gke_dl-platform_us-central1-b_platformv2/KubernetesUnregisteredCustomResourceCachingAgent[1/1] completed with one or more failures\r\n \r\nE  Exception in thread \"Exec Stream Pumper\" java.lang.OutOfMemoryError: Java heap space\r\n \r\nE  Exception in thread \"Exec Stream Pumper\" java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nE  Exception in thread \"Exec Stream Pumper\" java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n \r\nI  \r\n \r\nI  2018-09-04 18:15:18.347 ERROR 1 --- [gentScheduler-1] c.n.s.c.r.c.ClusteredAgentScheduler      : Unable to run agents\r\n \r\nI  \r\n \r\nI  java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n```\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/spinnaker/spinnaker/issues/comments/433177297","html_url":"https://github.com/spinnaker/spinnaker/issues/3287#issuecomment-433177297","issue_url":"https://api.github.com/repos/spinnaker/spinnaker/issues/3287","id":433177297,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzE3NzI5Nw==","user":{"login":"lwander","id":4874941,"node_id":"MDQ6VXNlcjQ4NzQ5NDE=","avatar_url":"https://avatars3.githubusercontent.com/u/4874941?v=4","gravatar_id":"","url":"https://api.github.com/users/lwander","html_url":"https://github.com/lwander","followers_url":"https://api.github.com/users/lwander/followers","following_url":"https://api.github.com/users/lwander/following{/other_user}","gists_url":"https://api.github.com/users/lwander/gists{/gist_id}","starred_url":"https://api.github.com/users/lwander/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lwander/subscriptions","organizations_url":"https://api.github.com/users/lwander/orgs","repos_url":"https://api.github.com/users/lwander/repos","events_url":"https://api.github.com/users/lwander/events{/privacy}","received_events_url":"https://api.github.com/users/lwander/received_events","type":"User","site_admin":false},"created_at":"2018-10-25T19:32:33Z","updated_at":"2018-10-25T19:32:33Z","author_association":"MEMBER","body":"Heh\r\n\r\nI believe @benjaminws might be seeing something similar -- just spoke to him about my diving in to fix memory usage. Does this ring any bells @benjaminws?"}}
{"actor":{"display_login":"lwander"}}