{"action":"created","issue":{"url":"https://api.github.com/repos/pingcap/tidb-operator/issues/1299","repository_url":"https://api.github.com/repos/pingcap/tidb-operator","labels_url":"https://api.github.com/repos/pingcap/tidb-operator/issues/1299/labels{/name}","comments_url":"https://api.github.com/repos/pingcap/tidb-operator/issues/1299/comments","events_url":"https://api.github.com/repos/pingcap/tidb-operator/issues/1299/events","html_url":"https://github.com/pingcap/tidb-operator/issues/1299","id":534341360,"node_id":"MDU6SXNzdWU1MzQzNDEzNjA=","number":1299,"title":"tidb node  upgrade error","user":{"login":"mikechengwei","id":13257127,"node_id":"MDQ6VXNlcjEzMjU3MTI3","avatar_url":"https://avatars2.githubusercontent.com/u/13257127?v=4","gravatar_id":"","url":"https://api.github.com/users/mikechengwei","html_url":"https://github.com/mikechengwei","followers_url":"https://api.github.com/users/mikechengwei/followers","following_url":"https://api.github.com/users/mikechengwei/following{/other_user}","gists_url":"https://api.github.com/users/mikechengwei/gists{/gist_id}","starred_url":"https://api.github.com/users/mikechengwei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikechengwei/subscriptions","organizations_url":"https://api.github.com/users/mikechengwei/orgs","repos_url":"https://api.github.com/users/mikechengwei/repos","events_url":"https://api.github.com/users/mikechengwei/events{/privacy}","received_events_url":"https://api.github.com/users/mikechengwei/received_events","type":"User","site_admin":false},"labels":[{"id":1017180920,"node_id":"MDU6TGFiZWwxMDE3MTgwOTIw","url":"https://api.github.com/repos/pingcap/tidb-operator/labels/type/question","name":"type/question","color":"d876e3","default":false,"description":"Further information is requested"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2019-12-07T02:29:15Z","updated_at":"2019-12-09T04:54:10Z","closed_at":null,"author_association":"NONE","body":"## Bug Report\r\n\r\n**What version of Kubernetes are you using?**\r\nv1.14.8\r\n\r\n**What version of TiDB Operator are you using?**\r\n1.0.4\r\n**What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?**\r\npd-ssd-tidb-cluster-pd-0                                            Bound    local-pv-ca55edf8   6249Gi     RWO            default        10d\r\npd-ssd-tidb-cluster-pd-1                                            Bound    local-pv-49045197   6254Gi     RWO            default        10d\r\npd-ssd-tidb-cluster-pd-2                                            Bound    local-pv-fd530dd6   6254Gi     RWO            default        10d\r\ntikv-ssd-tidb-cluster-tikv-0                                        Bound    local-pv-714e7b1d   6249Gi     RWO            default        10d\r\ntikv-ssd-tidb-cluster-tikv-1                                        Bound    local-pv-99768f9b   6249Gi     RWO            default        10d\r\ntikv-ssd-tidb-cluster-tikv-2                                        Bound    local-pv-59cf9d04   6254Gi     RWO            default        10d\r\n\r\n**What's the status of the TiDB cluster pods?**\r\nssd-tidb-cluster-pd-0                              1/1     Running            0          10d\r\nssd-tidb-cluster-pd-1                              1/1     Running            0          10d\r\nssd-tidb-cluster-pd-2                              1/1     Running            0          10d\r\nssd-tidb-cluster-tidb-0                            1/2     CrashLoopBackOff   11         36m\r\nssd-tidb-cluster-tidb-1                            1/2     CrashLoopBackOff   10         30m\r\nssd-tidb-cluster-tidb-2                            1/2     CrashLoopBackOff   10         30m\r\nssd-tidb-cluster-tikv-0                            1/1     Running            0          77m\r\nssd-tidb-cluster-tikv-1                            1/1     Running            0          78m\r\nssd-tidb-cluster-tikv-2                            1/1     Running            0          78m\r\n\r\n**What did you do?**\r\nAt the beginning, pd node and tikv node start with host mode,the tidb start with L2 mode.\r\nThen, I change the tidb network mode to host,which generate some error.\r\n\r\nFirstly,the number of tidb nodes is one，I change to  host network and replicas update to two.\r\nThis process I use kubectl tool to delete pod. \r\nNow,the number of  tidb nodes is three,it is not corrospond of expect replicas number. \r\n\r\nThe  helm history below.\r\n```\r\n5       \tThu Nov 28 23:13:10 2019\tsuperseded\ttidb-cluster-dev\t           \tUpgrade complete\r\n6       \tSat Dec  7 08:39:44 2019\tsuperseded\ttidb-cluster-dev\t           \tUpgrade complete\r\n7       \tSat Dec  7 08:40:03 2019\tsuperseded\ttidb-cluster-dev\t           \tUpgrade complete\r\n8       \tSat Dec  7 09:17:35 2019\tsuperseded\ttidb-cluster-dev\t           \tUpgrade complete\r\n9       \tSat Dec  7 09:21:09 2019\tsuperseded\ttidb-cluster-dev\t           \tUpgrade complete\r\n10      \tSat Dec  7 09:27:31 2019\tsuperseded\ttidb-cluster-dev\t           \tUpgrade complete\r\n11      \tSat Dec  7 09:50:53 2019\tdeployed  \ttidb-cluster-dev\t           \tUpgrade complete\r\n```\r\n```\r\nhelm get values --revision=5  ssd-tidb-cluster\r\ntdib:\r\n  config: |\r\n    [log]\r\n    level = \"info\"\r\n  hostNetwork: false\r\n  image: */pingcap/tidb:v3.0.5\r\n  imagePullPolicy: IfNotPresent\r\n  initializer:\r\n    resources: {}\r\n  maxFailoverCount: 3\r\n  nodeSelector: {}\r\n  plugin:\r\n    directory: /plugins\r\n    enable: false\r\n    list:\r\n    - whitelist-1\r\n  podSecurityContext: {}\r\n  replicas: 1\r\n```\r\n```\r\nhelm get values --revision=6  ssd-tidb-cluster (this revision I use kubectl tool to delete pod)\r\n tidb:\r\n  replicas: 2\r\n  resources:\r\n    limits:\r\n      cpu: \"12\"\r\n      memory: 16Gi\r\n    requests:\r\n      cpu: \"16\"\r\n      memory: 16Gi\r\n```\r\n```\r\n helm get values --revision=8  ssd-tidb-cluster\r\ntidb:\r\n  replicas: 1\r\n  resources:\r\n    limits:\r\n      cpu: \"12\"\r\n      memory: 16Gi\r\n    requests:\r\n      cpu: \"12\"\r\n      memory: 16Gi\r\n\r\n```\r\n```\r\nhelm get values --revision=11  ssd-tidb-cluster\r\ntidb:\r\n  replicas: 1\r\n  resources:\r\n    limits:\r\n      cpu: \"12\"\r\n      memory: 16Gi\r\n    requests:\r\n      cpu: \"12\"\r\n      memory: 16Gi\r\n```\r\nThis cluster current status is \r\n<img width=\"784\" alt=\"Screen Shot 2019-12-07 at 10 54 28 AM\" src=\"https://user-images.githubusercontent.com/13257127/70367905-f6772b00-18df-11ea-90fe-cb8580661925.png\">\r\n\r\n\r\nAnd the tidb can‘t start with log below.\r\n```\r\nstart tidb-server ...\r\n/tidb-server --store=tikv --host=0.0.0.0 --path=ssd-tidb-cluster-pd:2379 --config=/etc/tidb/tidb.toml\r\n --log-slow-query=/var/log/tidb/slowlog\r\n[2019/12/07 02:24:44.635 +00:00] [INFO] [printer.go:41] [\"Welcome to TiDB.\"] [\"Release Version\"=v3.0.5] [\"Git Commit Hash\"=c9000abdc216b6a02efbcc578af8be1f98ba280d] [\"Git Branch\"=HEAD] [\"UTC Build Time\"=\"2019-10-25 03:25:32\"] [GoVersion=\"go version go1.13 linux/amd64\"] [\"Race Enabled\"=false] [\"Check Table Before Drop\"=false] [\"TiKV Min Version\"=v3.0.0-60965b006877ca7234adaced7890d7b029ed1306]\r\n[2019/12/07 02:24:44.636 +00:00] [INFO] [printer.go:54] [\"loaded config\"] [config=\"{\\\"host\\\":\\\"0.0.0.0\\\",\\\"advertise-address\\\":\\\"\\\",\\\"port\\\":4000,\\\"cors\\\":\\\"\\\",\\\"store\\\":\\\"tikv\\\",\\\"path\\\":\\\"ssd-tidb-cluster-pd:2379\\\",\\\"socket\\\":\\\"\\\",\\\"lease\\\":\\\"45s\\\",\\\"run-ddl\\\":true,\\\"split-table\\\":true,\\\"token-limit\\\":1000,\\\"oom-action\\\":\\\"log\\\",\\\"mem-quota-query\\\":34359738368,\\\"enable-streaming\\\":false,\\\"txn-local-latches\\\":{\\\"enabled\\\":false,\\\"capacity\\\":2048000},\\\"lower-case-table-names\\\":2,\\\"log\\\":{\\\"level\\\":\\\"info\\\",\\\"format\\\":\\\"text\\\",\\\"disable-timestamp\\\":false,\\\"file\\\":{\\\"filename\\\":\\\"\\\",\\\"log-rotate\\\":true,\\\"max-size\\\":300,\\\"max-days\\\":0,\\\"max-backups\\\":0},\\\"slow-query-file\\\":\\\"/var/log/tidb/slowlog\\\",\\\"slow-threshold\\\":300,\\\"expensive-threshold\\\":10000,\\\"query-log-max-len\\\":2048,\\\"record-plan-in-slow-log\\\":1},\\\"security\\\":{\\\"skip-grant-table\\\":false,\\\"ssl-ca\\\":\\\"\\\",\\\"ssl-cert\\\":\\\"\\\",\\\"ssl-key\\\":\\\"\\\",\\\"cluster-ssl-ca\\\":\\\"\\\",\\\"cluster-ssl-cert\\\":\\\"\\\",\\\"cluster-ssl-key\\\":\\\"\\\"},\\\"status\\\":{\\\"report-status\\\":true,\\\"status-host\\\":\\\"0.0.0.0\\\",\\\"status-port\\\":10080,\\\"metrics-addr\\\":\\\"\\\",\\\"metrics-interval\\\":15,\\\"record-db-qps\\\":false},\\\"performance\\\":{\\\"max-procs\\\":0,\\\"max-memory\\\":0,\\\"tcp-keep-alive\\\":true,\\\"cross-join\\\":true,\\\"stats-lease\\\":\\\"3s\\\",\\\"run-auto-analyze\\\":true,\\\"stmt-count-limit\\\":5000,\\\"feedback-probability\\\":0.05,\\\"query-feedback-limit\\\":1024,\\\"pseudo-estimate-ratio\\\":0.8,\\\"force-priority\\\":\\\"NO_PRIORITY\\\",\\\"bind-info-lease\\\":\\\"3s\\\"},\\\"prepared-plan-cache\\\":{\\\"enabled\\\":false,\\\"capacity\\\":100,\\\"memory-guard-ratio\\\":0.1},\\\"opentracing\\\":{\\\"enable\\\":false,\\\"sampler\\\":{\\\"type\\\":\\\"const\\\",\\\"param\\\":1,\\\"sampling-server-url\\\":\\\"\\\",\\\"max-operations\\\":0,\\\"sampling-refresh-interval\\\":0},\\\"reporter\\\":{\\\"queue-size\\\":0,\\\"buffer-flush-interval\\\":0,\\\"log-spans\\\":false,\\\"local-agent-host-port\\\":\\\"\\\"},\\\"rpc-metrics\\\":false},\\\"proxy-protocol\\\":{\\\"networks\\\":\\\"\\\",\\\"header-timeout\\\":5},\\\"tikv-client\\\":{\\\"grpc-connection-count\\\":16,\\\"grpc-keepalive-time\\\":10,\\\"grpc-keepalive-timeout\\\":3,\\\"commit-timeout\\\":\\\"41s\\\",\\\"max-txn-time-use\\\":590,\\\"max-batch-size\\\":128,\\\"overload-threshold\\\":200,\\\"max-batch-wait-time\\\":0,\\\"batch-wait-size\\\":8,\\\"region-cache-ttl\\\":600},\\\"binlog\\\":{\\\"enable\\\":false,\\\"write-timeout\\\":\\\"15s\\\",\\\"ignore-error\\\":false,\\\"binlog-socket\\\":\\\"\\\",\\\"strategy\\\":\\\"range\\\"},\\\"compatible-kill-query\\\":false,\\\"plugin\\\":{\\\"dir\\\":\\\"\\\",\\\"load\\\":\\\"\\\"},\\\"pessimistic-txn\\\":{\\\"enable\\\":true,\\\"max-retry-count\\\":256,\\\"ttl\\\":\\\"40s\\\"},\\\"check-mb4-value-in-utf8\\\":true,\\\"treat-old-version-utf8-as-utf8mb4\\\":true,\\\"split-region-max-num\\\":1000,\\\"stmt-summary\\\":{\\\"max-stmt-count\\\":100,\\\"max-sql-length\\\":4096}}\"]\r\n[2019/12/07 02:24:44.636 +00:00] [INFO] [main.go:250] [\"disable Prometheus push client\"]\r\n[2019/12/07 02:24:44.636 +00:00] [INFO] [store.go:69] [\"new store\"] [path=tikv://ssd-tidb-cluster-pd:2379]\r\n[2019/12/07 02:24:44.636 +00:00] [INFO] [client.go:144] [\"[pd] create pd client with endpoints\"] [pd-address=\"[ssd-tidb-cluster-pd:2379]\"]\r\n[2019/12/07 02:24:44.636 +00:00] [INFO] [systime_mon.go:26] [\"start system time monitor\"]\r\n[2019/12/07 02:24:44.642 +00:00] [INFO] [client.go:252] [\"[pd] switch leader\"] [new-leader=http://ssd-tidb-cluster-pd-1.ssd-tidb-cluster-pd-peer.default.svc:2379] [old-leader=]\r\n[2019/12/07 02:24:44.642 +00:00] [INFO] [client.go:163] [\"[pd] init cluster id\"] [cluster-id=6763543047642807909]\r\n[2019/12/07 02:24:44.649 +00:00] [INFO] [store.go:75] [\"new store with retry success\"]\r\n[2019/12/07 02:25:06.404 +00:00] [WARN] [backoff.go:304] [\"pdRPC backoffer.maxSleep 20000ms is exceeded, errors:\\nloadRegion from PD failed, key: \\\"mBootstra\\\\xffpKey\\\\x00\\\\x00\\\\x00\\\\x00\\\\xfb\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00s\\\", err: invalid marker byte, group bytes \\\"furion#oh\\\" at 2019-12-07T02:25:01.021303864Z\\nloadRegion from PD failed, key: \\\"mBootstra\\\\xffpKey\\\\x00\\\\x00\\\\x00\\\\x00\\\\xfb\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00s\\\", err: invalid marker byte, group bytes \\\"furion#oh\\\" at 2019-12-07T02:25:03.586216772Z\\nloadRegion from PD failed, key: \\\"mBootstra\\\\xffpKey\\\\x00\\\\x00\\\\x00\\\\x00\\\\xfb\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00s\\\", err: invalid marker byte, group bytes \\\"furion#oh\\\" at 2019-12-07T02:25:06.404261693Z\"]\r\n[2019/12/07 02:25:06.404 +00:00] [FATAL] [session.go:1636] [\"check bootstrapped failed\"] [error=\"[tikv:9001]PD server timeout\"] [errorVerbose=\"[tikv:9001]PD server timeout\\ngithub.com/pingcap/errors.AddStack\\n\\tgithub.com/pingcap/errors@v0.11.4/errors.go:174\\ngithub.com/pingcap/errors.Trace\\n\\tgithub.com/pingcap/errors@v0.11.4/juju_adaptor.go:15\\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).loadRegion\\n\\tgithub.com/pingcap/tidb@/store/tikv/region_cache.go:604\\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).findRegionByKey\\n\\tgithub.com/pingcap/tidb@/store/tikv/region_cache.go:358\\ngithub.com/pingcap/tidb/store/tikv.(*RegionCache).LocateKey\\n\\tgithub.com/pingcap/tidb@/store/tikv/region_cache.go:318\\ngithub.com/pingcap/tidb/store/tikv.(*tikvSnapshot).get\\n\\tgithub.com/pingcap/tidb@/store/tikv/snapshot.go:301\\ngithub.com/pingcap/tidb/store/tikv.(*tikvSnapshot).Get\\n\\tgithub.com/pingcap/tidb@/store/tikv/snapshot.go:265\\ngithub.com/pingcap/tidb/kv.(*unionStore).Get\\n\\tgithub.com/pingcap/tidb@/kv/union_store.go:194\\ngithub.com/pingcap/tidb/store/tikv.(*tikvTxn).Get\\n\\tgithub.com/pingcap/tidb@/store/tikv/txn.go:133\\ngithub.com/pingcap/tidb/structure.(*TxStructure).Get\\n\\tgithub.com/pingcap/tidb@/structure/string.go:35\\ngithub.com/pingcap/tidb/structure.(*TxStructure).GetInt64\\n\\tgithub.com/pingcap/tidb@/structure/string.go:44\\ngithub.com/pingcap/tidb/meta.(*Meta).GetBootstrapVersion\\n\\tgithub.com/pingcap/tidb@/meta/meta.go:697\\ngithub.com/pingcap/tidb/session.getStoreBootstrapVersion.func1\\n\\tgithub.com/pingcap/tidb@/session/session.go:1631\\ngithub.com/pingcap/tidb/kv.RunInNewTxn\\n\\tgithub.com/pingcap/tidb@/kv/txn.go:50\\ngithub.com/pingcap/tidb/session.getStoreBootstrapVersion\\n\\tgithub.com/pingcap/tidb@/session/session.go:1628\\ngithub.com/pingcap/tidb/session.BootstrapSession\\n\\tgithub.com/pingcap/tidb@/session/session.go:1469\\nmain.createStoreAndDomain\\n\\tgithub.com/pingcap/tidb@/tidb-server/main.go:205\\nmain.main\\n\\tgithub.com/pingcap/tidb@/tidb-server/main.go:171\\nruntime.main\\n\\truntime/proc.go:203\\nruntime.goexit\\n\\truntime/asm_amd64.s:1357\"] [stack=\"github.com/pingcap/tidb/session.getStoreBootstrapVersion\\n\\tgithub.com/pingcap/tidb@/session/session.go:1636\\ngithub.com/pingcap/tidb/session.BootstrapSession\\n\\tgithub.com/pingcap/tidb@/session/session.go:1469\\nmain.createStoreAndDomain\\n\\tgithub.com/pingcap/tidb@/tidb-server/main.go:205\\nmain.main\\n\\tgithub.com/pingcap/tidb@/tidb-server/main.go:171\\nruntime.main\\n\\truntime/proc.go:203\"]\r\n```\r\n\r\n\r\n**What did you expect to see?**\r\nthe tidb start success\r\n**What did you see instead?**\r\nstart error\r\n"},"comment":{"url":"https://api.github.com/repos/pingcap/tidb-operator/issues/comments/563062440","html_url":"https://github.com/pingcap/tidb-operator/issues/1299#issuecomment-563062440","issue_url":"https://api.github.com/repos/pingcap/tidb-operator/issues/1299","id":563062440,"node_id":"MDEyOklzc3VlQ29tbWVudDU2MzA2MjQ0MA==","user":{"login":"mikechengwei","id":13257127,"node_id":"MDQ6VXNlcjEzMjU3MTI3","avatar_url":"https://avatars2.githubusercontent.com/u/13257127?v=4","gravatar_id":"","url":"https://api.github.com/users/mikechengwei","html_url":"https://github.com/mikechengwei","followers_url":"https://api.github.com/users/mikechengwei/followers","following_url":"https://api.github.com/users/mikechengwei/following{/other_user}","gists_url":"https://api.github.com/users/mikechengwei/gists{/gist_id}","starred_url":"https://api.github.com/users/mikechengwei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikechengwei/subscriptions","organizations_url":"https://api.github.com/users/mikechengwei/orgs","repos_url":"https://api.github.com/users/mikechengwei/repos","events_url":"https://api.github.com/users/mikechengwei/events{/privacy}","received_events_url":"https://api.github.com/users/mikechengwei/received_events","type":"User","site_admin":false},"created_at":"2019-12-09T04:54:10Z","updated_at":"2019-12-09T04:54:10Z","author_association":"NONE","body":"I tried,and can‘t working.I think operator can't work well in  the pod's failover state. The condition is pods are healthy,right?\r\n"}}
{"actor":{"display_login":"mikechengwei"}}