{"action":"created","issue":{"url":"https://api.github.com/repos/pingcap/tidb/issues/8171","repository_url":"https://api.github.com/repos/pingcap/tidb","labels_url":"https://api.github.com/repos/pingcap/tidb/issues/8171/labels{/name}","comments_url":"https://api.github.com/repos/pingcap/tidb/issues/8171/comments","events_url":"https://api.github.com/repos/pingcap/tidb/issues/8171/events","html_url":"https://github.com/pingcap/tidb/pull/8171","id":377286992,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI4MjY3ODE1","number":8171,"title":"ddl: fix a bug that cancel add index ddl job when workers are not run, which cause tidb ddl hang up","user":{"login":"winkyao","id":6949944,"node_id":"MDQ6VXNlcjY5NDk5NDQ=","avatar_url":"https://avatars0.githubusercontent.com/u/6949944?v=4","gravatar_id":"","url":"https://api.github.com/users/winkyao","html_url":"https://github.com/winkyao","followers_url":"https://api.github.com/users/winkyao/followers","following_url":"https://api.github.com/users/winkyao/following{/other_user}","gists_url":"https://api.github.com/users/winkyao/gists{/gist_id}","starred_url":"https://api.github.com/users/winkyao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/winkyao/subscriptions","organizations_url":"https://api.github.com/users/winkyao/orgs","repos_url":"https://api.github.com/users/winkyao/repos","events_url":"https://api.github.com/users/winkyao/events{/privacy}","received_events_url":"https://api.github.com/users/winkyao/received_events","type":"User","site_admin":false},"labels":[{"id":745133021,"node_id":"MDU6TGFiZWw3NDUxMzMwMjE=","url":"https://api.github.com/repos/pingcap/tidb/labels/component/DDL-need-LGT3","name":"component/DDL-need-LGT3","color":"00bff4","default":false},{"id":630229257,"node_id":"MDU6TGFiZWw2MzAyMjkyNTc=","url":"https://api.github.com/repos/pingcap/tidb/labels/type/bug-fix","name":"type/bug-fix","color":"fc2929","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-05T07:59:33Z","updated_at":"2018-11-06T12:21:03Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/pingcap/tidb/pulls/8171","html_url":"https://github.com/pingcap/tidb/pull/8171","diff_url":"https://github.com/pingcap/tidb/pull/8171.diff","patch_url":"https://github.com/pingcap/tidb/pull/8171.patch"},"body":"\r\n<!--\r\nThank you for contributing to TiDB! Please read TiDB's [CONTRIBUTING](https://github.com/pingcap/tidb/blob/master/CONTRIBUTING.md) document **BEFORE** filing this PR.\r\n-->\r\n\r\n### What problem does this PR solve? <!--add issue link with summary if exists-->\r\n\r\nIf we cancel add index ddl job when the `job.Type == model.ActionAddIndex && job.SchemaState == model.StateWriteReorganization && job.SnapshotVer == 0`, the TiDB ddl will hang up like:\r\n\r\n```\r\nMySQL [(none)]> admin show ddl jobs;                                                                                                                                                   [11/973]\r\n+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+                                           \r\n| JOBS                                                                                                                                                                                                                                | STATE     |                                           \r\n+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+                                           \r\n| ID:230, Type:drop index, State:running, SchemaState:none, SchemaID:31, TableID:48, RowCount:0, ArgLen:0, start time: 2018-10-25 17:24:31.123 +0800 CST, Err:[ddl:101]invalid table state public, ErrCount:344905, SnapshotVersion:0 | running   |                                           \r\n| ID:231, Type:drop index, State:none, SchemaState:none, SchemaID:31, TableID:51, RowCount:0, ArgLen:0, start time: 2018-10-25 17:27:10.173 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0                                       | none      |                                           \r\n| ID:232, Type:drop index, State:none, SchemaState:none, SchemaID:31, TableID:54, RowCount:0, ArgLen:0, start time: 2018-10-25 17:27:11.273 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0                                       | none      |                                           \r\n```\r\n\r\n### What is changed and how it works?\r\n\r\n```\r\n// runDDLJob runs a DDL job. It returns the current schema version in this transaction and the error.\r\nfunc (w *worker) runDDLJob(d *ddlCtx, t *meta.Meta, job *model.Job) (ver int64, err error) {\r\n   log.Infof(\"[ddl-%s] run DDL job %s\", w, job)\r\n   timeStart := time.Now()\r\n   defer func() {\r\n      metrics.DDLWorkerHistogram.WithLabelValues(metrics.WorkerRunDDLJob, job.Type.String(), metrics.RetLabel(err)).Observe(time.Since(timeStart).Seconds())\r\n   }()\r\n   if job.IsFinished() {\r\n      return\r\n   }\r\n   // The cause of this job state is that the job is cancelled by client.\r\n   if job.IsCancelling() {\r\n      // If the value of SnapshotVer isn't zero, it means the work is backfilling the indexes.\r\n      if job.Type == model.ActionAddIndex && job.SchemaState == model.StateWriteReorganization && job.SnapshotVer != 0 {\r\n         log.Infof(\"[ddl-%s] run the cancelling DDL job %s\", w, job)\r\n         w.reorgCtx.notifyReorgCancel()\r\n      } else {\r\n         job.State = model.JobStateCancelled\r\n         job.Error = errCancelledDDLJob\r\n         job.ErrorCount++\r\n         return\r\n      }\r\n   }\r\n```\r\n\r\nif `job.Type == model.ActionAddIndex && job.SchemaState == model.StateWriteReorganization && job.SnapshotVer == 0`, we will set job.State to model.JobStateCancelled, unfortunately, the adding index state is model.StateWriteReorganization, and it will not be rollbacked. So when we execute drop index, we will got error:\r\n\r\n```\r\ntidb-2018-10-25T19-14-23.472.log:2018/10/25 17:44:44.356 ddl_worker.go:329: [info] [ddl] run DDL job ID:230, Type:drop index, State:running, SchemaState:none, SchemaID:31, TableID:48, RowCount:0, ArgLen:0, start time: 2018-10-25 17:24:31.123 +0800 CST, Err:[ddl:101]invalid table state write reorganization, ErrCount:172588, SnapshotVersion:0\r\n```\r\nit is returned by:\r\n\r\n```\r\nfunc onDropIndex(t *meta.Meta, job *model.Job) (ver int64, _ error) {\r\n\t...\r\n\r\n\toriginalState := indexInfo.State\r\n\tswitch indexInfo.State {\r\n\tcase model.StatePublic:\r\n                ...\r\n\tcase model.StateWriteOnly:\r\n\t\t...\r\n\tcase model.StateDeleteOnly:\r\n\t\t...\r\n\tcase model.StateDeleteReorganization:\r\n\t\t...\r\n\tdefault:\r\n\t\terr = ErrInvalidTableState.GenWithStack(\"invalid table state %v\", tblInfo.State)\r\n\t}\r\n```\r\n\r\n\r\n### Check List <!--REMOVE the items that are not applicable-->\r\n\r\nTests <!-- At least one of them must be included. -->\r\n\r\n - Unit test\r\n\r\n\r\nRelated changes\r\n\r\n - Need to cherry-pick to the release branch\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/pingcap/tidb/issues/comments/436233874","html_url":"https://github.com/pingcap/tidb/pull/8171#issuecomment-436233874","issue_url":"https://api.github.com/repos/pingcap/tidb/issues/8171","id":436233874,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjIzMzg3NA==","user":{"login":"crazycs520","id":26020263,"node_id":"MDQ6VXNlcjI2MDIwMjYz","avatar_url":"https://avatars2.githubusercontent.com/u/26020263?v=4","gravatar_id":"","url":"https://api.github.com/users/crazycs520","html_url":"https://github.com/crazycs520","followers_url":"https://api.github.com/users/crazycs520/followers","following_url":"https://api.github.com/users/crazycs520/following{/other_user}","gists_url":"https://api.github.com/users/crazycs520/gists{/gist_id}","starred_url":"https://api.github.com/users/crazycs520/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crazycs520/subscriptions","organizations_url":"https://api.github.com/users/crazycs520/orgs","repos_url":"https://api.github.com/users/crazycs520/repos","events_url":"https://api.github.com/users/crazycs520/events{/privacy}","received_events_url":"https://api.github.com/users/crazycs520/received_events","type":"User","site_admin":false},"created_at":"2018-11-06T12:21:03Z","updated_at":"2018-11-06T12:21:03Z","author_association":"CONTRIBUTOR","body":"/run-all-tests"}}
{"actor":{"display_login":"crazycs520"}}