{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/3969","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/3969/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/3969/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/3969/events","html_url":"https://github.com/rancher/rancher/issues/3969","id":140843459,"number":3969,"title":"upgrade from 0.59.0 -> 0.63.1-rc1 -> Exception relating to service reconcile seen in logs.","user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"},{"url":"https://api.github.com/repos/rancher/rancher/labels/status/to-test","name":"status/to-test","color":"66aa66"}],"state":"closed","locked":false,"assignee":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/4","html_url":"https://github.com/rancher/rancher/milestones/Release%201.0","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/4/labels","id":947265,"number":4,"title":"Release 1.0","description":"1.0 GA","creator":{"login":"will-chan","id":10134676,"avatar_url":"https://avatars.githubusercontent.com/u/10134676?v=3","gravatar_id":"","url":"https://api.github.com/users/will-chan","html_url":"https://github.com/will-chan","followers_url":"https://api.github.com/users/will-chan/followers","following_url":"https://api.github.com/users/will-chan/following{/other_user}","gists_url":"https://api.github.com/users/will-chan/gists{/gist_id}","starred_url":"https://api.github.com/users/will-chan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-chan/subscriptions","organizations_url":"https://api.github.com/users/will-chan/orgs","repos_url":"https://api.github.com/users/will-chan/repos","events_url":"https://api.github.com/users/will-chan/events{/privacy}","received_events_url":"https://api.github.com/users/will-chan/received_events","type":"User","site_admin":false},"open_issues":289,"closed_issues":371,"state":"open","created_at":"2015-01-22T18:32:24Z","updated_at":"2016-03-16T00:48:53Z","due_on":"2016-03-31T07:00:00Z","closed_at":null},"comments":2,"created_at":"2016-03-15T01:26:28Z","updated_at":"2016-03-16T00:48:53Z","closed_at":"2016-03-16T00:48:53Z","body":"Setup - upgrade from 0.59.0 -> 0.63.1-rc1 with few services.\r\n\r\nAfter upgrade , following exception seen in logs for every service:\r\n\r\nService itself is OK and is in good state. I dont see instances of service being recreated. \r\n\r\n```\r\n2016-03-15 00:58:56,984 ERROR [:] [] [] [] [ecutorService-7] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [service.update:494] on [10] java.lang.IllegalStateException: Failed to do service reconcile for service [10]\r\n\tat io.cattle.platform.servicediscovery.deployment.impl.DeploymentManagerImpl$1.doWithLock(DeploymentManagerImpl.java:135) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.servicediscovery.deployment.impl.DeploymentManagerImpl$1.doWithLock(DeploymentManagerImpl.java:109) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.servicediscovery.deployment.impl.DeploymentManagerImpl.activate(DeploymentManagerImpl.java:109) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.servicediscovery.deployment.impl.DeploymentManagerImpl.activate(DeploymentManagerImpl.java:92) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.servicediscovery.process.ServiceUpdateActivate.handle(ServiceUpdateActivate.java:77) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:440) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:387) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:381) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:381) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:487) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:314) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:254) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:251) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:183) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:156) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:106) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:103) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:103) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:68) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:54) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:76) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:72) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]\r\n```"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/197089077","html_url":"https://github.com/rancher/rancher/issues/3969#issuecomment-197089077","issue_url":"https://api.github.com/repos/rancher/rancher/issues/3969","id":197089077,"user":{"login":"sangeethah","id":4266958,"avatar_url":"https://avatars.githubusercontent.com/u/4266958?v=3","gravatar_id":"","url":"https://api.github.com/users/sangeethah","html_url":"https://github.com/sangeethah","followers_url":"https://api.github.com/users/sangeethah/followers","following_url":"https://api.github.com/users/sangeethah/following{/other_user}","gists_url":"https://api.github.com/users/sangeethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sangeethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sangeethah/subscriptions","organizations_url":"https://api.github.com/users/sangeethah/orgs","repos_url":"https://api.github.com/users/sangeethah/repos","events_url":"https://api.github.com/users/sangeethah/events{/privacy}","received_events_url":"https://api.github.com/users/sangeethah/received_events","type":"User","site_admin":false},"created_at":"2016-03-16T00:48:53Z","updated_at":"2016-03-16T00:48:53Z","body":"Service reconcile being done as part of upgrade and reporting failure is not an issue by itself. The only thing that we want to address here is that stack traces  are not being logged when service reconcile fails and it should be logged as \"INFO\" rather than \"ERROR\" . This issue is already being tracked in\r\nhttps://github.com/rancher/rancher/issues/3370. Closing this issue.\r\n"}}