{"action":"created","issue":{"url":"https://api.github.com/repos/rmosolgo/graphql-ruby/issues/1776","repository_url":"https://api.github.com/repos/rmosolgo/graphql-ruby","labels_url":"https://api.github.com/repos/rmosolgo/graphql-ruby/issues/1776/labels{/name}","comments_url":"https://api.github.com/repos/rmosolgo/graphql-ruby/issues/1776/comments","events_url":"https://api.github.com/repos/rmosolgo/graphql-ruby/issues/1776/events","html_url":"https://github.com/rmosolgo/graphql-ruby/pull/1776","id":350770478,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA4NTMzOTYz","number":1776,"title":"fix QUOTED_STRING and BLOCK_STRING tokenizing problem","user":{"login":"tompng","id":1780201,"node_id":"MDQ6VXNlcjE3ODAyMDE=","avatar_url":"https://avatars3.githubusercontent.com/u/1780201?v=4","gravatar_id":"","url":"https://api.github.com/users/tompng","html_url":"https://github.com/tompng","followers_url":"https://api.github.com/users/tompng/followers","following_url":"https://api.github.com/users/tompng/following{/other_user}","gists_url":"https://api.github.com/users/tompng/gists{/gist_id}","starred_url":"https://api.github.com/users/tompng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tompng/subscriptions","organizations_url":"https://api.github.com/users/tompng/orgs","repos_url":"https://api.github.com/users/tompng/repos","events_url":"https://api.github.com/users/tompng/events{/privacy}","received_events_url":"https://api.github.com/users/tompng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-15T11:06:42Z","updated_at":"2018-08-17T03:52:20Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/rmosolgo/graphql-ruby/pulls/1776","html_url":"https://github.com/rmosolgo/graphql-ruby/pull/1776","diff_url":"https://github.com/rmosolgo/graphql-ruby/pull/1776.diff","patch_url":"https://github.com/rmosolgo/graphql-ruby/pull/1776.patch"},"body":"QUOTED_STRING and BLOCK_STRING does not seem to be tokenized correctly.\r\n\r\n```ruby\r\nGraphQL::VERSION\r\n# => \"1.8.7\"\r\n\r\n# QUOTED_STRING\r\nGraphQL::Language::Lexer.run_lexer '\"\\\\\\\\\", \"\\\\\"\", \"\\\\\"\\\\\\\\\"'\r\n# => [(STRING \"\\\\\\\", \" [1:1]), (UNKNOWN_CHAR \"\\\\\" [1:7]), (STRING \"\" [1:8]), (STRING \"\\\"\\\\\" [1:11])]\r\n\r\n# BLOCK_STRING\r\nGraphQL::Language::Lexer.run_lexer %|\"\"\"\"a\"\"\",\"\"\"b\"\"\"|\r\n# => [(STRING \"\\\"a\\\"\\\"\\\", \\\"\\\"\\\"b\" [1:1])] (should be [(STRING \"a\"), (STRING \"b\")]\r\n\r\nGraphQL::Language::Lexer.run_lexer %|\"\"\"\\\\n\"\"\"|\r\n# => [(STRING \"\\n\" [1:1])] (should be \"\\\\\" + \"n\". should not be unescaped)\r\n\r\nGraphQL::Language::Lexer.run_lexer %|\"\"\"\"\"\"|\r\nGraphQL::Language::Lexer.run_lexer %|\"\"\"\\n\\n\\n\"\"\"|\r\n# => NoMethodError: undefined method `empty?' for nil:NilClass\r\n```\r\n\r\nThis pullrequest:\r\n- fixes definition of QUOTED_STRING and BLOCK_STRING\r\n- fixes NoMethodError when tokenizing empty BLOCK_STRING\r\n\r\n# Quoted String\r\nquoted string consists of escaped characters(`\\` + any) and normal characters(not `\\\"`)\r\n```\r\nBACKSLASH_ESCAPED_CHAR = '\\\\' /./;\r\nSTRING_CHAR =   (BACKSLASH_ESCAPED_CHAR | [^\\\\\"]);\r\n```\r\n\r\n# Block String\r\nany string that does not end with `\"` or ends with `\"\"\"` can be explicitly splitted into these 3 groups.\r\nBLOCK_QUOTE `\"\"\"`\r\nBLOCK_STRING_ESCAPE_FRAGMENT `\\+` `\"\\+` `\"\"\\+`\r\nBLOCK_STRING_FRAGMENT `[^\\\"]+` `\"[^\\\"]+` `\"\"[^\\\"]+`\r\n\r\nand the restrictions are:\r\nBLOCK_QUOTE should appear after BLOCK_STRING_ESCAPE_FRAGMENT\r\nshould not end with BLOCK_STRING_ESCAPE_FRAGMENT\r\n```\r\nSHORT_BLOCK_QUOTES = ('' | '\"' | '\"\"');\r\nBLOCK_STRING_FRAGMENT = SHORT_BLOCK_QUOTES [^\\\\\"]+;\r\nBLOCK_STRING_ESCAPE_FRAGMENT = SHORT_BLOCK_QUOTES '\\\\'+;\r\nBLOCK_STRING_CHAR = (BLOCK_STRING_ESCAPE_FRAGMENT+ '\"\"\"' | BLOCK_STRING_ESCAPE_FRAGMENT* BLOCK_STRING_FRAGMENT);\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/rmosolgo/graphql-ruby/issues/comments/413750215","html_url":"https://github.com/rmosolgo/graphql-ruby/pull/1776#issuecomment-413750215","issue_url":"https://api.github.com/repos/rmosolgo/graphql-ruby/issues/1776","id":413750215,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzc1MDIxNQ==","user":{"login":"tompng","id":1780201,"node_id":"MDQ6VXNlcjE3ODAyMDE=","avatar_url":"https://avatars3.githubusercontent.com/u/1780201?v=4","gravatar_id":"","url":"https://api.github.com/users/tompng","html_url":"https://github.com/tompng","followers_url":"https://api.github.com/users/tompng/followers","following_url":"https://api.github.com/users/tompng/following{/other_user}","gists_url":"https://api.github.com/users/tompng/gists{/gist_id}","starred_url":"https://api.github.com/users/tompng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tompng/subscriptions","organizations_url":"https://api.github.com/users/tompng/orgs","repos_url":"https://api.github.com/users/tompng/repos","events_url":"https://api.github.com/users/tompng/events{/privacy}","received_events_url":"https://api.github.com/users/tompng/received_events","type":"User","site_admin":false},"created_at":"2018-08-17T03:52:20Z","updated_at":"2018-08-17T03:52:20Z","author_association":"NONE","body":"```ruby\r\ns = '\"\"\"' + '\\\\\"\"\"\"\"' * 1000 + '\\\\\"\"'\r\nGraphQL::Language::Lexer.tokenize s\r\n```\r\n\r\nwill be tokenized to\r\n`\"\"`, `\"\\\"\"`, `\"\"`, `\"\\\"\"`, ..., `\"\"`, `\"\\\"\"`, `\"\"`\r\n(there is no UNKNOWN_CHAR, but it takes very long time)\r\n\r\nMaybe adding UNMATCHED_BLOCK_QUOTE and UNMATCHED_QUOTED_STRING will solve this performance problem.\r\n(If it starts with '\"\"\"', it will always be BLOCK_QUOTE or UNMATCHED_BLOCK_QUOTE)\r\n```\r\nQUOTED_STRING = QUOTE STRING_CHAR* QUOTE;\r\nBLOCK_STRING = BLOCK_QUOTE BLOCK_STRING_CHAR* BLOCK_QUOTE;\r\nUNMATCHED_QUOTED_STRING = QUOTE STRING_CHAR*;\r\nUNMATCHED_BLOCK_STRING = BLOCK_QUOTE BLOCK_STRING_CHAR* SHORT_BLOCK_QUOTES;\r\n\r\nUNMATCHED_BLOCK_STRING => { emit(:UNMATCHED_BLOCK_STRING, ts, te, meta) };\r\nUNMATCHED_QUOTED_STRING => { emit(:UNMATCHED_QUOTED_STRING, ts, te, meta) };\r\n```\r\n(I will update this pull request later)"}}
{"actor":{"display_login":"tompng"}}