{"action":"closed","number":711,"pull_request":{"url":"https://api.github.com/repos/pingcap/parser/pulls/711","id":359697911,"node_id":"MDExOlB1bGxSZXF1ZXN0MzU5Njk3OTEx","html_url":"https://github.com/pingcap/parser/pull/711","diff_url":"https://github.com/pingcap/parser/pull/711.diff","patch_url":"https://github.com/pingcap/parser/pull/711.patch","issue_url":"https://api.github.com/repos/pingcap/parser/issues/711","number":711,"state":"closed","locked":false,"title":"Rewrite special comment parser","user":{"login":"kennytm","id":103023,"node_id":"MDQ6VXNlcjEwMzAyMw==","avatar_url":"https://avatars1.githubusercontent.com/u/103023?v=4","gravatar_id":"","url":"https://api.github.com/users/kennytm","html_url":"https://github.com/kennytm","followers_url":"https://api.github.com/users/kennytm/followers","following_url":"https://api.github.com/users/kennytm/following{/other_user}","gists_url":"https://api.github.com/users/kennytm/gists{/gist_id}","starred_url":"https://api.github.com/users/kennytm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kennytm/subscriptions","organizations_url":"https://api.github.com/users/kennytm/orgs","repos_url":"https://api.github.com/users/kennytm/repos","events_url":"https://api.github.com/users/kennytm/events{/privacy}","received_events_url":"https://api.github.com/users/kennytm/received_events","type":"User","site_admin":false},"body":"<!--\r\nThank you for contributing to TiDB SQL Parser! Please read [this](https://github.com/pingcap/parser/blob/master/README.md) document **BEFORE** filing this PR.\r\n-->\r\n\r\n### What problem does this PR solve? <!--add issue link with summary if exists-->\r\n\r\nRewrite the special comment parser (`/*! ... */` and `/*+ ... */`) to better match MySQL's behavior. \r\n\r\nThis is the first step to solve https://github.com/pingcap/tidb/issues/14030#issuecomment-567791327.\r\n\r\n### What is changed and how it works?\r\n\r\n* Removed `specialCommentScanner`. Instead:\r\n    1. When encountering `/*!` or `/*!12345` or `/*T!` or `/*T!012345`, drop this substring and continue parsing as if outside a comment. Use a flag to indicate we should skip the stray `*/`.\r\n    2. When encountering `/*+`, read the entire comment as a string token. This token will be parsed into `[]*ast.TableOptimizerHint` using the new *hint parser* on-demand. All errors coming from that parsers are demoted to warnings.\r\n\r\n* The `/*+ ... */` is only parsed after keywords SELECT, INSERT, REPLACE, UPDATE, DELETE, CREATE and PARTITION, otherwise it is treated as an ordinary comment. Therefore, some SQL previously failing like `SELECT 1 FROM /*+ hash_agg() */ t1` now becomes successful and ignoring the hint. \r\n\r\n    * Unlike MySQL, however, statements like `SELECT 1 FOR UPDATE /*+ max_execution_time(60) */` would still produce an error. We could fix it later?\r\n\r\n* And we now have two parsers, the main `parser.y`, and the optimizer-hint specific `hintparser.y`\r\n\r\n    1. All hint-related rules from deleted from `parser.y`. This also deletes the related TiDB-specific keywords.\r\n    2. goyacc is modified to support generating two parsers. Also make goyacc to exit with error on conflict, so we don't need to run the parser twice on the same .y file.\r\n    3. In the new hint parser, all optimizer hints from MySQL 5.7+ are recognized for completeness (though all of them are ignored with warnings).\r\n\r\n* The structure `ast.TableOptimizerHint` is *not yet changed*, so we are still API-compatible.\r\n\r\n### Check List <!--REMOVE the items that are not applicable-->\r\n\r\nTests <!-- At least one of them must be included. -->\r\n\r\n - Unit test\r\n\r\nCode changes\r\n\r\nSide effects\r\n\r\n - Possible performance regression\r\n - Breaking backward compatibility\r\n\r\nRelated changes\r\n\r\n - Need to be included in the release note\r\n","created_at":"2020-01-06T20:19:06Z","updated_at":"2020-01-08T06:55:51Z","closed_at":"2020-01-08T06:55:51Z","merged_at":"2020-01-08T06:55:51Z","merge_commit_sha":"7de2c9958237d536b3f8a632474bedc204a66a95","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":1117649034,"node_id":"MDU6TGFiZWwxMTE3NjQ5MDM0","url":"https://api.github.com/repos/pingcap/parser/labels/status/LGT2","name":"status/LGT2","color":"5319e7","default":false,"description":"LGT2"},{"id":1484368514,"node_id":"MDU6TGFiZWwxNDg0MzY4NTE0","url":"https://api.github.com/repos/pingcap/parser/labels/status/PTAL","name":"status/PTAL","color":"d876e3","default":false,"description":"PTAL"},{"id":1097891261,"node_id":"MDU6TGFiZWwxMDk3ODkxMjYx","url":"https://api.github.com/repos/pingcap/parser/labels/type/enhancement","name":"type/enhancement","color":"a2eeef","default":false,"description":"New feature or request"}],"milestone":null,"commits_url":"https://api.github.com/repos/pingcap/parser/pulls/711/commits","review_comments_url":"https://api.github.com/repos/pingcap/parser/pulls/711/comments","review_comment_url":"https://api.github.com/repos/pingcap/parser/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pingcap/parser/issues/711/comments","statuses_url":"https://api.github.com/repos/pingcap/parser/statuses/9de90a3e8def915e105aa3d0af04935768152443","head":{"label":"pingcap:kennytm/revamp-special-comments","ref":"kennytm/revamp-special-comments","sha":"9de90a3e8def915e105aa3d0af04935768152443","user":{"login":"pingcap","id":11855343,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExODU1MzQz","avatar_url":"https://avatars3.githubusercontent.com/u/11855343?v=4","gravatar_id":"","url":"https://api.github.com/users/pingcap","html_url":"https://github.com/pingcap","followers_url":"https://api.github.com/users/pingcap/followers","following_url":"https://api.github.com/users/pingcap/following{/other_user}","gists_url":"https://api.github.com/users/pingcap/gists{/gist_id}","starred_url":"https://api.github.com/users/pingcap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pingcap/subscriptions","organizations_url":"https://api.github.com/users/pingcap/orgs","repos_url":"https://api.github.com/users/pingcap/repos","events_url":"https://api.github.com/users/pingcap/events{/privacy}","received_events_url":"https://api.github.com/users/pingcap/received_events","type":"Organization","site_admin":false},"repo":{"id":153882884,"node_id":"MDEwOlJlcG9zaXRvcnkxNTM4ODI4ODQ=","name":"parser","full_name":"pingcap/parser","private":false,"owner":{"login":"pingcap","id":11855343,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExODU1MzQz","avatar_url":"https://avatars3.githubusercontent.com/u/11855343?v=4","gravatar_id":"","url":"https://api.github.com/users/pingcap","html_url":"https://github.com/pingcap","followers_url":"https://api.github.com/users/pingcap/followers","following_url":"https://api.github.com/users/pingcap/following{/other_user}","gists_url":"https://api.github.com/users/pingcap/gists{/gist_id}","starred_url":"https://api.github.com/users/pingcap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pingcap/subscriptions","organizations_url":"https://api.github.com/users/pingcap/orgs","repos_url":"https://api.github.com/users/pingcap/repos","events_url":"https://api.github.com/users/pingcap/events{/privacy}","received_events_url":"https://api.github.com/users/pingcap/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/pingcap/parser","description":"TiDB SQL Parser","fork":false,"url":"https://api.github.com/repos/pingcap/parser","forks_url":"https://api.github.com/repos/pingcap/parser/forks","keys_url":"https://api.github.com/repos/pingcap/parser/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pingcap/parser/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pingcap/parser/teams","hooks_url":"https://api.github.com/repos/pingcap/parser/hooks","issue_events_url":"https://api.github.com/repos/pingcap/parser/issues/events{/number}","events_url":"https://api.github.com/repos/pingcap/parser/events","assignees_url":"https://api.github.com/repos/pingcap/parser/assignees{/user}","branches_url":"https://api.github.com/repos/pingcap/parser/branches{/branch}","tags_url":"https://api.github.com/repos/pingcap/parser/tags","blobs_url":"https://api.github.com/repos/pingcap/parser/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pingcap/parser/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pingcap/parser/git/refs{/sha}","trees_url":"https://api.github.com/repos/pingcap/parser/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pingcap/parser/statuses/{sha}","languages_url":"https://api.github.com/repos/pingcap/parser/languages","stargazers_url":"https://api.github.com/repos/pingcap/parser/stargazers","contributors_url":"https://api.github.com/repos/pingcap/parser/contributors","subscribers_url":"https://api.github.com/repos/pingcap/parser/subscribers","subscription_url":"https://api.github.com/repos/pingcap/parser/subscription","commits_url":"https://api.github.com/repos/pingcap/parser/commits{/sha}","git_commits_url":"https://api.github.com/repos/pingcap/parser/git/commits{/sha}","comments_url":"https://api.github.com/repos/pingcap/parser/comments{/number}","issue_comment_url":"https://api.github.com/repos/pingcap/parser/issues/comments{/number}","contents_url":"https://api.github.com/repos/pingcap/parser/contents/{+path}","compare_url":"https://api.github.com/repos/pingcap/parser/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pingcap/parser/merges","archive_url":"https://api.github.com/repos/pingcap/parser/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pingcap/parser/downloads","issues_url":"https://api.github.com/repos/pingcap/parser/issues{/number}","pulls_url":"https://api.github.com/repos/pingcap/parser/pulls{/number}","milestones_url":"https://api.github.com/repos/pingcap/parser/milestones{/number}","notifications_url":"https://api.github.com/repos/pingcap/parser/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pingcap/parser/labels{/name}","releases_url":"https://api.github.com/repos/pingcap/parser/releases{/id}","deployments_url":"https://api.github.com/repos/pingcap/parser/deployments","created_at":"2018-10-20T07:55:25Z","updated_at":"2020-01-07T09:50:16Z","pushed_at":"2020-01-08T06:55:51Z","git_url":"git://github.com/pingcap/parser.git","ssh_url":"git@github.com:pingcap/parser.git","clone_url":"https://github.com/pingcap/parser.git","svn_url":"https://github.com/pingcap/parser","homepage":null,"size":28273,"stargazers_count":460,"watchers_count":460,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":226,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":40,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":226,"open_issues":40,"watchers":460,"default_branch":"master"}},"base":{"label":"pingcap:master","ref":"master","sha":"18bb5e80b59c727789f67a8085ae5601e6180eef","user":{"login":"pingcap","id":11855343,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExODU1MzQz","avatar_url":"https://avatars3.githubusercontent.com/u/11855343?v=4","gravatar_id":"","url":"https://api.github.com/users/pingcap","html_url":"https://github.com/pingcap","followers_url":"https://api.github.com/users/pingcap/followers","following_url":"https://api.github.com/users/pingcap/following{/other_user}","gists_url":"https://api.github.com/users/pingcap/gists{/gist_id}","starred_url":"https://api.github.com/users/pingcap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pingcap/subscriptions","organizations_url":"https://api.github.com/users/pingcap/orgs","repos_url":"https://api.github.com/users/pingcap/repos","events_url":"https://api.github.com/users/pingcap/events{/privacy}","received_events_url":"https://api.github.com/users/pingcap/received_events","type":"Organization","site_admin":false},"repo":{"id":153882884,"node_id":"MDEwOlJlcG9zaXRvcnkxNTM4ODI4ODQ=","name":"parser","full_name":"pingcap/parser","private":false,"owner":{"login":"pingcap","id":11855343,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExODU1MzQz","avatar_url":"https://avatars3.githubusercontent.com/u/11855343?v=4","gravatar_id":"","url":"https://api.github.com/users/pingcap","html_url":"https://github.com/pingcap","followers_url":"https://api.github.com/users/pingcap/followers","following_url":"https://api.github.com/users/pingcap/following{/other_user}","gists_url":"https://api.github.com/users/pingcap/gists{/gist_id}","starred_url":"https://api.github.com/users/pingcap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pingcap/subscriptions","organizations_url":"https://api.github.com/users/pingcap/orgs","repos_url":"https://api.github.com/users/pingcap/repos","events_url":"https://api.github.com/users/pingcap/events{/privacy}","received_events_url":"https://api.github.com/users/pingcap/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/pingcap/parser","description":"TiDB SQL Parser","fork":false,"url":"https://api.github.com/repos/pingcap/parser","forks_url":"https://api.github.com/repos/pingcap/parser/forks","keys_url":"https://api.github.com/repos/pingcap/parser/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pingcap/parser/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pingcap/parser/teams","hooks_url":"https://api.github.com/repos/pingcap/parser/hooks","issue_events_url":"https://api.github.com/repos/pingcap/parser/issues/events{/number}","events_url":"https://api.github.com/repos/pingcap/parser/events","assignees_url":"https://api.github.com/repos/pingcap/parser/assignees{/user}","branches_url":"https://api.github.com/repos/pingcap/parser/branches{/branch}","tags_url":"https://api.github.com/repos/pingcap/parser/tags","blobs_url":"https://api.github.com/repos/pingcap/parser/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pingcap/parser/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pingcap/parser/git/refs{/sha}","trees_url":"https://api.github.com/repos/pingcap/parser/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pingcap/parser/statuses/{sha}","languages_url":"https://api.github.com/repos/pingcap/parser/languages","stargazers_url":"https://api.github.com/repos/pingcap/parser/stargazers","contributors_url":"https://api.github.com/repos/pingcap/parser/contributors","subscribers_url":"https://api.github.com/repos/pingcap/parser/subscribers","subscription_url":"https://api.github.com/repos/pingcap/parser/subscription","commits_url":"https://api.github.com/repos/pingcap/parser/commits{/sha}","git_commits_url":"https://api.github.com/repos/pingcap/parser/git/commits{/sha}","comments_url":"https://api.github.com/repos/pingcap/parser/comments{/number}","issue_comment_url":"https://api.github.com/repos/pingcap/parser/issues/comments{/number}","contents_url":"https://api.github.com/repos/pingcap/parser/contents/{+path}","compare_url":"https://api.github.com/repos/pingcap/parser/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pingcap/parser/merges","archive_url":"https://api.github.com/repos/pingcap/parser/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pingcap/parser/downloads","issues_url":"https://api.github.com/repos/pingcap/parser/issues{/number}","pulls_url":"https://api.github.com/repos/pingcap/parser/pulls{/number}","milestones_url":"https://api.github.com/repos/pingcap/parser/milestones{/number}","notifications_url":"https://api.github.com/repos/pingcap/parser/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pingcap/parser/labels{/name}","releases_url":"https://api.github.com/repos/pingcap/parser/releases{/id}","deployments_url":"https://api.github.com/repos/pingcap/parser/deployments","created_at":"2018-10-20T07:55:25Z","updated_at":"2020-01-07T09:50:16Z","pushed_at":"2020-01-08T06:55:51Z","git_url":"git://github.com/pingcap/parser.git","ssh_url":"git@github.com:pingcap/parser.git","clone_url":"https://github.com/pingcap/parser.git","svn_url":"https://github.com/pingcap/parser","homepage":null,"size":28273,"stargazers_count":460,"watchers_count":460,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":226,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":40,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":226,"open_issues":40,"watchers":460,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pingcap/parser/pulls/711"},"html":{"href":"https://github.com/pingcap/parser/pull/711"},"issue":{"href":"https://api.github.com/repos/pingcap/parser/issues/711"},"comments":{"href":"https://api.github.com/repos/pingcap/parser/issues/711/comments"},"review_comments":{"href":"https://api.github.com/repos/pingcap/parser/pulls/711/comments"},"review_comment":{"href":"https://api.github.com/repos/pingcap/parser/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pingcap/parser/pulls/711/commits"},"statuses":{"href":"https://api.github.com/repos/pingcap/parser/statuses/9de90a3e8def915e105aa3d0af04935768152443"}},"author_association":"MEMBER","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"kennytm","id":103023,"node_id":"MDQ6VXNlcjEwMzAyMw==","avatar_url":"https://avatars1.githubusercontent.com/u/103023?v=4","gravatar_id":"","url":"https://api.github.com/users/kennytm","html_url":"https://github.com/kennytm","followers_url":"https://api.github.com/users/kennytm/followers","following_url":"https://api.github.com/users/kennytm/following{/other_user}","gists_url":"https://api.github.com/users/kennytm/gists{/gist_id}","starred_url":"https://api.github.com/users/kennytm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kennytm/subscriptions","organizations_url":"https://api.github.com/users/kennytm/orgs","repos_url":"https://api.github.com/users/kennytm/repos","events_url":"https://api.github.com/users/kennytm/events{/privacy}","received_events_url":"https://api.github.com/users/kennytm/received_events","type":"User","site_admin":false},"comments":4,"review_comments":2,"maintainer_can_modify":false,"commits":12,"additions":10663,"deletions":9195,"changed_files":17}}
{"actor":{"display_login":"kennytm"}}