{"action":"created","comment":{"url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/comments/91135895","pull_request_review_id":11645253,"id":91135895,"diff_hunk":"@@ -239,34 +269,27 @@ static Tree *parser__lex(Parser *self, TSStateId parse_state) {\n         self->lexer.data.result_symbol = ts_builtin_sym_error;\n         break;\n       }\n-      self->lexer.data.advance(&self->lexer, ERROR_STATE, false);\n+      self->lexer.data.advance(&self->lexer, false);\n     }\n \n     skipped_error = true;\n     error_end_position = self->lexer.current_position;\n   }\n \n   Tree *result;\n-\n   if (skipped_error) {\n     Length padding = length_sub(error_start_position, start_position);\n     Length size = length_sub(error_end_position, error_start_position);\n     ts_lexer_reset(&self->lexer, error_end_position);\n     result = ts_tree_make_error(size, padding, first_error_character);\n   } else {\n     TSSymbol symbol = self->lexer.data.result_symbol;\n-    Length padding =\n-      length_sub(self->lexer.token_start_position, start_position);\n-    Length size = length_sub(self->lexer.current_position,\n-                                  self->lexer.token_start_position);\n-    result =\n-      ts_tree_make_leaf(symbol, padding, size,\n-                        ts_language_symbol_metadata(self->language, symbol));\n+    Length padding = length_sub(self->lexer.token_start_position, start_position);\n+    Length size = length_sub(self->lexer.current_position, self->lexer.token_start_position);\n+    TSSymbolMetadata metadata = ts_language_symbol_metadata(self->language, symbol);\n+    result = ts_tree_make_leaf(symbol, padding, size, metadata);\n   }\n \n-  if (!result)","path":"src/runtime/parser.c","position":107,"original_position":107,"commit_id":"83514293b50971e766edb8cae2351824303399a7","original_commit_id":"83514293b50971e766edb8cae2351824303399a7","user":{"login":"maxbrunsfeld","id":326587,"avatar_url":"https://avatars.githubusercontent.com/u/326587?v=3","gravatar_id":"","url":"https://api.github.com/users/maxbrunsfeld","html_url":"https://github.com/maxbrunsfeld","followers_url":"https://api.github.com/users/maxbrunsfeld/followers","following_url":"https://api.github.com/users/maxbrunsfeld/following{/other_user}","gists_url":"https://api.github.com/users/maxbrunsfeld/gists{/gist_id}","starred_url":"https://api.github.com/users/maxbrunsfeld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxbrunsfeld/subscriptions","organizations_url":"https://api.github.com/users/maxbrunsfeld/orgs","repos_url":"https://api.github.com/users/maxbrunsfeld/repos","events_url":"https://api.github.com/users/maxbrunsfeld/events{/privacy}","received_events_url":"https://api.github.com/users/maxbrunsfeld/received_events","type":"User","site_admin":true},"body":"Yeah, this conditional was leftover from before https://github.com/tree-sitter/tree-sitter/pull/43, when the system tried to recover from malloc failure. I've removed that logic across the board, since malloc never even actually returns null on most systems, and apps that use tree-sitter aren't likely to have a fine-grained OOM recovery strategy anyway.","created_at":"2016-12-06T18:05:02Z","updated_at":"2016-12-06T18:05:02Z","html_url":"https://github.com/tree-sitter/tree-sitter/pull/54#discussion_r91135895","pull_request_url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54","_links":{"self":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/comments/91135895"},"html":{"href":"https://github.com/tree-sitter/tree-sitter/pull/54#discussion_r91135895"},"pull_request":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54"}}},"pull_request":{"url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54","id":96390093,"html_url":"https://github.com/tree-sitter/tree-sitter/pull/54","diff_url":"https://github.com/tree-sitter/tree-sitter/pull/54.diff","patch_url":"https://github.com/tree-sitter/tree-sitter/pull/54.patch","issue_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/54","number":54,"state":"open","locked":false,"title":"External scanners","user":{"login":"maxbrunsfeld","id":326587,"avatar_url":"https://avatars.githubusercontent.com/u/326587?v=3","gravatar_id":"","url":"https://api.github.com/users/maxbrunsfeld","html_url":"https://github.com/maxbrunsfeld","followers_url":"https://api.github.com/users/maxbrunsfeld/followers","following_url":"https://api.github.com/users/maxbrunsfeld/following{/other_user}","gists_url":"https://api.github.com/users/maxbrunsfeld/gists{/gist_id}","starred_url":"https://api.github.com/users/maxbrunsfeld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxbrunsfeld/subscriptions","organizations_url":"https://api.github.com/users/maxbrunsfeld/orgs","repos_url":"https://api.github.com/users/maxbrunsfeld/repos","events_url":"https://api.github.com/users/maxbrunsfeld/events{/privacy}","received_events_url":"https://api.github.com/users/maxbrunsfeld/received_events","type":"User","site_admin":true},"body":"This adds the ability for a tree-sitter parser to tokenize input using arbitrary hand-written code. \r\n\r\nThe API for external scanners will need to grow and/or change in order to allow the scanner to maintain its state properly during incremental parsing, but I think it's worth shipping this with only batch parsing support, so that we can try it out with the Ruby grammar, and maybe start work on a Python grammar.","created_at":"2016-12-03T07:30:41Z","updated_at":"2016-12-06T18:05:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"f31c907b0c5083a3178814270f400060bc439d2b","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54/commits","review_comments_url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54/comments","review_comment_url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/comments{/number}","comments_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/54/comments","statuses_url":"https://api.github.com/repos/tree-sitter/tree-sitter/statuses/83514293b50971e766edb8cae2351824303399a7","head":{"label":"tree-sitter:external-scanners","ref":"external-scanners","sha":"83514293b50971e766edb8cae2351824303399a7","user":{"login":"tree-sitter","id":17203512,"avatar_url":"https://avatars.githubusercontent.com/u/17203512?v=3","gravatar_id":"","url":"https://api.github.com/users/tree-sitter","html_url":"https://github.com/tree-sitter","followers_url":"https://api.github.com/users/tree-sitter/followers","following_url":"https://api.github.com/users/tree-sitter/following{/other_user}","gists_url":"https://api.github.com/users/tree-sitter/gists{/gist_id}","starred_url":"https://api.github.com/users/tree-sitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tree-sitter/subscriptions","organizations_url":"https://api.github.com/users/tree-sitter/orgs","repos_url":"https://api.github.com/users/tree-sitter/repos","events_url":"https://api.github.com/users/tree-sitter/events{/privacy}","received_events_url":"https://api.github.com/users/tree-sitter/received_events","type":"Organization","site_admin":false},"repo":{"id":14164618,"name":"tree-sitter","full_name":"tree-sitter/tree-sitter","owner":{"login":"tree-sitter","id":17203512,"avatar_url":"https://avatars.githubusercontent.com/u/17203512?v=3","gravatar_id":"","url":"https://api.github.com/users/tree-sitter","html_url":"https://github.com/tree-sitter","followers_url":"https://api.github.com/users/tree-sitter/followers","following_url":"https://api.github.com/users/tree-sitter/following{/other_user}","gists_url":"https://api.github.com/users/tree-sitter/gists{/gist_id}","starred_url":"https://api.github.com/users/tree-sitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tree-sitter/subscriptions","organizations_url":"https://api.github.com/users/tree-sitter/orgs","repos_url":"https://api.github.com/users/tree-sitter/repos","events_url":"https://api.github.com/users/tree-sitter/events{/privacy}","received_events_url":"https://api.github.com/users/tree-sitter/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/tree-sitter/tree-sitter","description":"An incremental parsing system for programming tools","fork":false,"url":"https://api.github.com/repos/tree-sitter/tree-sitter","forks_url":"https://api.github.com/repos/tree-sitter/tree-sitter/forks","keys_url":"https://api.github.com/repos/tree-sitter/tree-sitter/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tree-sitter/tree-sitter/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tree-sitter/tree-sitter/teams","hooks_url":"https://api.github.com/repos/tree-sitter/tree-sitter/hooks","issue_events_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/events{/number}","events_url":"https://api.github.com/repos/tree-sitter/tree-sitter/events","assignees_url":"https://api.github.com/repos/tree-sitter/tree-sitter/assignees{/user}","branches_url":"https://api.github.com/repos/tree-sitter/tree-sitter/branches{/branch}","tags_url":"https://api.github.com/repos/tree-sitter/tree-sitter/tags","blobs_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/refs{/sha}","trees_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tree-sitter/tree-sitter/statuses/{sha}","languages_url":"https://api.github.com/repos/tree-sitter/tree-sitter/languages","stargazers_url":"https://api.github.com/repos/tree-sitter/tree-sitter/stargazers","contributors_url":"https://api.github.com/repos/tree-sitter/tree-sitter/contributors","subscribers_url":"https://api.github.com/repos/tree-sitter/tree-sitter/subscribers","subscription_url":"https://api.github.com/repos/tree-sitter/tree-sitter/subscription","commits_url":"https://api.github.com/repos/tree-sitter/tree-sitter/commits{/sha}","git_commits_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/commits{/sha}","comments_url":"https://api.github.com/repos/tree-sitter/tree-sitter/comments{/number}","issue_comment_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/comments{/number}","contents_url":"https://api.github.com/repos/tree-sitter/tree-sitter/contents/{+path}","compare_url":"https://api.github.com/repos/tree-sitter/tree-sitter/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tree-sitter/tree-sitter/merges","archive_url":"https://api.github.com/repos/tree-sitter/tree-sitter/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tree-sitter/tree-sitter/downloads","issues_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues{/number}","pulls_url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls{/number}","milestones_url":"https://api.github.com/repos/tree-sitter/tree-sitter/milestones{/number}","notifications_url":"https://api.github.com/repos/tree-sitter/tree-sitter/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tree-sitter/tree-sitter/labels{/name}","releases_url":"https://api.github.com/repos/tree-sitter/tree-sitter/releases{/id}","deployments_url":"https://api.github.com/repos/tree-sitter/tree-sitter/deployments","created_at":"2013-11-06T06:16:00Z","updated_at":"2016-11-15T03:54:20Z","pushed_at":"2016-12-06T01:26:17Z","git_url":"git://github.com/tree-sitter/tree-sitter.git","ssh_url":"git@github.com:tree-sitter/tree-sitter.git","clone_url":"https://github.com/tree-sitter/tree-sitter.git","svn_url":"https://github.com/tree-sitter/tree-sitter","homepage":"","size":6993,"stargazers_count":44,"watchers_count":44,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":6,"forks":6,"open_issues":6,"watchers":44,"default_branch":"master"}},"base":{"label":"tree-sitter:master","ref":"master","sha":"269ee1c1207bc6b3ecf907e464d1f57e9e88e941","user":{"login":"tree-sitter","id":17203512,"avatar_url":"https://avatars.githubusercontent.com/u/17203512?v=3","gravatar_id":"","url":"https://api.github.com/users/tree-sitter","html_url":"https://github.com/tree-sitter","followers_url":"https://api.github.com/users/tree-sitter/followers","following_url":"https://api.github.com/users/tree-sitter/following{/other_user}","gists_url":"https://api.github.com/users/tree-sitter/gists{/gist_id}","starred_url":"https://api.github.com/users/tree-sitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tree-sitter/subscriptions","organizations_url":"https://api.github.com/users/tree-sitter/orgs","repos_url":"https://api.github.com/users/tree-sitter/repos","events_url":"https://api.github.com/users/tree-sitter/events{/privacy}","received_events_url":"https://api.github.com/users/tree-sitter/received_events","type":"Organization","site_admin":false},"repo":{"id":14164618,"name":"tree-sitter","full_name":"tree-sitter/tree-sitter","owner":{"login":"tree-sitter","id":17203512,"avatar_url":"https://avatars.githubusercontent.com/u/17203512?v=3","gravatar_id":"","url":"https://api.github.com/users/tree-sitter","html_url":"https://github.com/tree-sitter","followers_url":"https://api.github.com/users/tree-sitter/followers","following_url":"https://api.github.com/users/tree-sitter/following{/other_user}","gists_url":"https://api.github.com/users/tree-sitter/gists{/gist_id}","starred_url":"https://api.github.com/users/tree-sitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tree-sitter/subscriptions","organizations_url":"https://api.github.com/users/tree-sitter/orgs","repos_url":"https://api.github.com/users/tree-sitter/repos","events_url":"https://api.github.com/users/tree-sitter/events{/privacy}","received_events_url":"https://api.github.com/users/tree-sitter/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/tree-sitter/tree-sitter","description":"An incremental parsing system for programming tools","fork":false,"url":"https://api.github.com/repos/tree-sitter/tree-sitter","forks_url":"https://api.github.com/repos/tree-sitter/tree-sitter/forks","keys_url":"https://api.github.com/repos/tree-sitter/tree-sitter/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tree-sitter/tree-sitter/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tree-sitter/tree-sitter/teams","hooks_url":"https://api.github.com/repos/tree-sitter/tree-sitter/hooks","issue_events_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/events{/number}","events_url":"https://api.github.com/repos/tree-sitter/tree-sitter/events","assignees_url":"https://api.github.com/repos/tree-sitter/tree-sitter/assignees{/user}","branches_url":"https://api.github.com/repos/tree-sitter/tree-sitter/branches{/branch}","tags_url":"https://api.github.com/repos/tree-sitter/tree-sitter/tags","blobs_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/refs{/sha}","trees_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tree-sitter/tree-sitter/statuses/{sha}","languages_url":"https://api.github.com/repos/tree-sitter/tree-sitter/languages","stargazers_url":"https://api.github.com/repos/tree-sitter/tree-sitter/stargazers","contributors_url":"https://api.github.com/repos/tree-sitter/tree-sitter/contributors","subscribers_url":"https://api.github.com/repos/tree-sitter/tree-sitter/subscribers","subscription_url":"https://api.github.com/repos/tree-sitter/tree-sitter/subscription","commits_url":"https://api.github.com/repos/tree-sitter/tree-sitter/commits{/sha}","git_commits_url":"https://api.github.com/repos/tree-sitter/tree-sitter/git/commits{/sha}","comments_url":"https://api.github.com/repos/tree-sitter/tree-sitter/comments{/number}","issue_comment_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/comments{/number}","contents_url":"https://api.github.com/repos/tree-sitter/tree-sitter/contents/{+path}","compare_url":"https://api.github.com/repos/tree-sitter/tree-sitter/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tree-sitter/tree-sitter/merges","archive_url":"https://api.github.com/repos/tree-sitter/tree-sitter/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tree-sitter/tree-sitter/downloads","issues_url":"https://api.github.com/repos/tree-sitter/tree-sitter/issues{/number}","pulls_url":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls{/number}","milestones_url":"https://api.github.com/repos/tree-sitter/tree-sitter/milestones{/number}","notifications_url":"https://api.github.com/repos/tree-sitter/tree-sitter/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tree-sitter/tree-sitter/labels{/name}","releases_url":"https://api.github.com/repos/tree-sitter/tree-sitter/releases{/id}","deployments_url":"https://api.github.com/repos/tree-sitter/tree-sitter/deployments","created_at":"2013-11-06T06:16:00Z","updated_at":"2016-11-15T03:54:20Z","pushed_at":"2016-12-06T01:26:17Z","git_url":"git://github.com/tree-sitter/tree-sitter.git","ssh_url":"git@github.com:tree-sitter/tree-sitter.git","clone_url":"https://github.com/tree-sitter/tree-sitter.git","svn_url":"https://github.com/tree-sitter/tree-sitter","homepage":"","size":6993,"stargazers_count":44,"watchers_count":44,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":6,"forks":6,"open_issues":6,"watchers":44,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54"},"html":{"href":"https://github.com/tree-sitter/tree-sitter/pull/54"},"issue":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/54"},"comments":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/issues/54/comments"},"review_comments":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54/comments"},"review_comment":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/pulls/54/commits"},"statuses":{"href":"https://api.github.com/repos/tree-sitter/tree-sitter/statuses/83514293b50971e766edb8cae2351824303399a7"}}}}
{"actor":{"display_login":"maxbrunsfeld"}}