{"action":"closed","issue":{"url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/1395","repository_url":"https://api.github.com/repos/Azure/service-fabric-issues","labels_url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/1395/labels{/name}","comments_url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/1395/comments","events_url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/1395/events","html_url":"https://github.com/Azure/service-fabric-issues/issues/1395","id":392360982,"node_id":"MDU6SXNzdWUzOTIzNjA5ODI=","number":1395,"title":"Error upgrading the SF cluster to latest version (manual & automatic)","user":{"login":"Lemraj","id":10677560,"node_id":"MDQ6VXNlcjEwNjc3NTYw","avatar_url":"https://avatars2.githubusercontent.com/u/10677560?v=4","gravatar_id":"","url":"https://api.github.com/users/Lemraj","html_url":"https://github.com/Lemraj","followers_url":"https://api.github.com/users/Lemraj/followers","following_url":"https://api.github.com/users/Lemraj/following{/other_user}","gists_url":"https://api.github.com/users/Lemraj/gists{/gist_id}","starred_url":"https://api.github.com/users/Lemraj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lemraj/subscriptions","organizations_url":"https://api.github.com/users/Lemraj/orgs","repos_url":"https://api.github.com/users/Lemraj/repos","events_url":"https://api.github.com/users/Lemraj/events{/privacy}","received_events_url":"https://api.github.com/users/Lemraj/received_events","type":"User","site_admin":false},"labels":[{"id":365329777,"node_id":"MDU6TGFiZWwzNjUzMjk3Nzc=","url":"https://api.github.com/repos/Azure/service-fabric-issues/labels/question","name":"question","color":"cc317c","default":true}],"state":"closed","locked":false,"assignee":{"login":"maburlik","id":15791906,"node_id":"MDQ6VXNlcjE1NzkxOTA2","avatar_url":"https://avatars2.githubusercontent.com/u/15791906?v=4","gravatar_id":"","url":"https://api.github.com/users/maburlik","html_url":"https://github.com/maburlik","followers_url":"https://api.github.com/users/maburlik/followers","following_url":"https://api.github.com/users/maburlik/following{/other_user}","gists_url":"https://api.github.com/users/maburlik/gists{/gist_id}","starred_url":"https://api.github.com/users/maburlik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maburlik/subscriptions","organizations_url":"https://api.github.com/users/maburlik/orgs","repos_url":"https://api.github.com/users/maburlik/repos","events_url":"https://api.github.com/users/maburlik/events{/privacy}","received_events_url":"https://api.github.com/users/maburlik/received_events","type":"User","site_admin":false},"assignees":[{"login":"maburlik","id":15791906,"node_id":"MDQ6VXNlcjE1NzkxOTA2","avatar_url":"https://avatars2.githubusercontent.com/u/15791906?v=4","gravatar_id":"","url":"https://api.github.com/users/maburlik","html_url":"https://github.com/maburlik","followers_url":"https://api.github.com/users/maburlik/followers","following_url":"https://api.github.com/users/maburlik/following{/other_user}","gists_url":"https://api.github.com/users/maburlik/gists{/gist_id}","starred_url":"https://api.github.com/users/maburlik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maburlik/subscriptions","organizations_url":"https://api.github.com/users/maburlik/orgs","repos_url":"https://api.github.com/users/maburlik/repos","events_url":"https://api.github.com/users/maburlik/events{/privacy}","received_events_url":"https://api.github.com/users/maburlik/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-12-18T22:22:39Z","updated_at":"2019-01-02T19:19:51Z","closed_at":"2019-01-02T19:19:51Z","author_association":"NONE","body":"Hi,\r\n\r\nI have a question about upgrading the SF clusters. We have 3 SF clusters (Dev\\QA\\PRD), all set to automatic upgrade mode. The Dev cluster is upgraded to the latest version (**6.4.617.9590**) , the other clusters still have the version (**6.3.187.9494**). I discovered this after deploying an application (updated with the laterst SF packages) to the QA cluster. The new application was succefully deployed, but we got runtime errors.\r\n\r\nI tried to manually to upgrade the cluster to the latest version using the portal & ARM template, but I get now this error:\r\n\r\n`{\"code\":\"DeploymentFailed\",\"message\":\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.\",\"details\":[{\"code\":\"Conflict\",\"message\":\"{ \"status\": \"Failed\", \"error\": { \"code\": \"ResourceDeploymentFailure\", \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\", \"details\": [ { \"code\": \"ClusterUpgradeFailed\", \"message\": \"Cluster upgrade failed. Reason Code: 'HealthCheck', Upgrade Progress: '{\"upgradeDescription\":{\"targetCodeVersion\":\"6.3.187.9494\",\"targetConfigVersion\":\"55\",\"upgradePolicyDescription\":{\"upgradeMode\":\"UnmonitoredAuto\",\"forceRestart\":false,\"upgradeReplicaSetCheckTimeout\":\"00:20:00\",\"kind\":\"Rolling\"}},\"targetCodeVersion\":\"6.3.187.9494\",\"targetConfigVersion\":\"55\",\"upgradeState\":\"RollingBackCompleted\",\"upgradeDomains\":[{\"name\":\"0\",\"state\":\"Completed\"},{\"name\":\"1\",\"state\":\"Completed\"},{\"name\":\"2\",\"state\":\"Completed\"},{\"name\":\"3\",\"state\":\"Completed\"},{\"name\":\"4\",\"state\":\"Completed\"}],\"rollingUpgradeMode\":\"UnmonitoredAuto\",\"upgradeDuration\":\"00:49:03\",\"currentUpgradeDomainDuration\":\"00:00:00\",\"unhealthyEvaluations\":[{\"baselineErrorCount\":0,\"baselineTotalCount\":7,\"totalCount\":7,\"maxPercentDeltaUnhealthyNodes\":0,\"unhealthyEvaluations\":[{\"nodeName\":\"_Frontend_0\",\"unhealthyEvaluations\":[{\"unhealthyEvent\":{\"sourceUtcTimestamp\":\"2018-12-18T22:01:20.0181726Z\",\"lastModifiedUtcTimestamp\":\"2018-12-18T22:01:29.541081Z\",\"isExpired\":false,\"lastOkTransitionAt\":\"0001-01-01T00:00:00\",\"lastWarningTransitionAt\":\"0001-01-01T00:00:00\",\"lastErrorTransitionAt\":\"2018-12-18T22:01:29.541081Z\",\"healthInformation\":{\"sourceId\":\"System.FM\",\"property\":\"State\",\"healthState\":\"Error\",\"description\":\"Fabric node is down.\",\"timeToLive\":\"10675199.02:48:05.4775807\",\"removeWhenExpired\":false,\"sequenceNumber\":908}},\"considerWarningAsError\":false,\"kind\":\"Event\",\"description\":\"Error event: SourceId='System.FM', Property='State'.\",\"aggregatedHealthState\":\"Error\"}],\"kind\":\"Node\",\"description\":\"Unhealthy node: NodeName='_Frontend_0', AggregatedHealthState='Error'.\",\"aggregatedHealthState\":\"Error\"}],\"kind\":\"DeltaNodesCheck\",\"description\":\"Delta nodes health check failed. Baseline and current status: 0% (0/7) - 14% (1/7), MaxPercentDeltaUnhealthyNodes=0%.\",\"aggregatedHealthState\":\"Error\"}],\"currentUpgradeDomainProgress\":{\"upgradeDomainName\":\"\",\"nodeProgressList\":[]},\"startTimestampUtc\":\"2018-12-18T21:14:53.489004Z\",\"failureTimestampUtc\":\"2018-12-18T22:01:56.879326Z\",\"failureReason\":\"HealthCheck\",\"upgradeDomainProgressAtFailure\":{\"upgradeDomainName\":\"\",\"nodeProgressList\":[]}}'.\", \"details\": [] } ] }}\"}]}`\r\n\r\n---------\r\n**Errors in Event Viewer in VM node:**\r\nFaulting application name: Fabric.exe, version: 6.4.617.9590, time stamp: 0x5bf55547\r\nFaulting module name: CONCRT140.dll, version: 6.3.9600.19153, time stamp: 0x5b93ffa7\r\nException code: 0xc0000135\r\nFault offset: 0x00000000000ecf30\r\nFaulting process id: 0x878\r\nFaulting application start time: 0x01d496e8c6256620\r\nFaulting application path: C:\\Program Files\\Microsoft Service Fabric\\bin\\Fabric\\Fabric.Code\\Fabric.exe\r\nFaulting module path: CONCRT140.dll\r\nReport Id: 03d6b6cc-02dc-11e9-810f-000d3a27a9af\r\nFaulting package full name: \r\nFaulting package-relative application ID: \r\n----------\r\n\r\nAny idea why this is happening?\r\n\r\nThanks."}}
{"actor":{"display_login":"Lemraj"}}