{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/vscode-textmate/issues/15","repository_url":"https://api.github.com/repos/Microsoft/vscode-textmate","labels_url":"https://api.github.com/repos/Microsoft/vscode-textmate/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/vscode-textmate/issues/15/comments","events_url":"https://api.github.com/repos/Microsoft/vscode-textmate/issues/15/events","html_url":"https://github.com/Microsoft/vscode-textmate/issues/15","id":160700345,"number":15,"title":"Tokenizer fails when line is empty / newlines","user":{"login":"felixsanz","id":6182099,"avatar_url":"https://avatars.githubusercontent.com/u/6182099?v=3","gravatar_id":"","url":"https://api.github.com/users/felixsanz","html_url":"https://github.com/felixsanz","followers_url":"https://api.github.com/users/felixsanz/followers","following_url":"https://api.github.com/users/felixsanz/following{/other_user}","gists_url":"https://api.github.com/users/felixsanz/gists{/gist_id}","starred_url":"https://api.github.com/users/felixsanz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/felixsanz/subscriptions","organizations_url":"https://api.github.com/users/felixsanz/orgs","repos_url":"https://api.github.com/users/felixsanz/repos","events_url":"https://api.github.com/users/felixsanz/events{/privacy}","received_events_url":"https://api.github.com/users/felixsanz/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Microsoft/vscode-textmate/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":{"login":"alexandrudima","id":5047891,"avatar_url":"https://avatars.githubusercontent.com/u/5047891?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandrudima","html_url":"https://github.com/alexandrudima","followers_url":"https://api.github.com/users/alexandrudima/followers","following_url":"https://api.github.com/users/alexandrudima/following{/other_user}","gists_url":"https://api.github.com/users/alexandrudima/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandrudima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandrudima/subscriptions","organizations_url":"https://api.github.com/users/alexandrudima/orgs","repos_url":"https://api.github.com/users/alexandrudima/repos","events_url":"https://api.github.com/users/alexandrudima/events{/privacy}","received_events_url":"https://api.github.com/users/alexandrudima/received_events","type":"User","site_admin":false},"assignees":[{"login":"alexandrudima","id":5047891,"avatar_url":"https://avatars.githubusercontent.com/u/5047891?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandrudima","html_url":"https://github.com/alexandrudima","followers_url":"https://api.github.com/users/alexandrudima/followers","following_url":"https://api.github.com/users/alexandrudima/following{/other_user}","gists_url":"https://api.github.com/users/alexandrudima/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandrudima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandrudima/subscriptions","organizations_url":"https://api.github.com/users/alexandrudima/orgs","repos_url":"https://api.github.com/users/alexandrudima/repos","events_url":"https://api.github.com/users/alexandrudima/events{/privacy}","received_events_url":"https://api.github.com/users/alexandrudima/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/Microsoft/vscode-textmate/milestones/2","html_url":"https://github.com/Microsoft/vscode-textmate/milestones/Backlog","labels_url":"https://api.github.com/repos/Microsoft/vscode-textmate/milestones/2/labels","id":1944544,"number":2,"title":"Backlog","description":"","creator":{"login":"alexandrudima","id":5047891,"avatar_url":"https://avatars.githubusercontent.com/u/5047891?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandrudima","html_url":"https://github.com/alexandrudima","followers_url":"https://api.github.com/users/alexandrudima/followers","following_url":"https://api.github.com/users/alexandrudima/following{/other_user}","gists_url":"https://api.github.com/users/alexandrudima/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandrudima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandrudima/subscriptions","organizations_url":"https://api.github.com/users/alexandrudima/orgs","repos_url":"https://api.github.com/users/alexandrudima/repos","events_url":"https://api.github.com/users/alexandrudima/events{/privacy}","received_events_url":"https://api.github.com/users/alexandrudima/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":3,"state":"open","created_at":"2016-08-16T09:14:36Z","updated_at":"2016-10-22T13:00:18Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-06-16T16:09:16Z","updated_at":"2016-10-22T13:00:18Z","closed_at":"2016-10-22T13:00:18Z","body":"I don't know if i'm doing it correctly, but i'm actually doing:\r\n\r\n```javascript\r\nlet ruleStack\r\nconst tokens = snippet.split('\\n').reduce((tokenList, line) => {\r\n   const r = grammar.tokenizeLine(line, ruleStack)\r\n   ruleStack = r.ruleStack\r\n```\r\n\r\nSo basically i split the piece of code by new line characters and tokenize line by line. The problem is that some lines contain just new line characters, for example an empty line that separate 2 lines of code like this:\r\n\r\n```javascript\r\nvar foo\r\n// <--- this line contains just \\n\r\nvar bar\r\n```\r\n\r\nBut since i'm splitting the new line characters to parse line by line, that line contains nothing now.\r\n\r\nBut the tokenizer still generates this:\r\n\r\n```\r\n{ tokens: [ { startIndex: 0, endIndex: 1, scopes: [Object] } ],\r\n  ruleStack: \r\n   [ StackElement {\r\n       ruleId: 1,\r\n       enterPos: -1,\r\n       endRule: null,\r\n       scopeName: 'source.js',\r\n       contentName: null,\r\n       whileRule: null } ] }\r\n```\r\n\r\nstartIndex 0 is ok, but why endIndex is 1 and not 0? Looks like there is a character but there is none.\r\n\r\nIf you add `line.charCodeAt(0)` you get NaN because there is nothing there. the same with `line.slice(0, 1)`"},"comment":{"url":"https://api.github.com/repos/Microsoft/vscode-textmate/issues/comments/255527072","html_url":"https://github.com/Microsoft/vscode-textmate/issues/15#issuecomment-255527072","issue_url":"https://api.github.com/repos/Microsoft/vscode-textmate/issues/15","id":255527072,"user":{"login":"alexandrudima","id":5047891,"avatar_url":"https://avatars.githubusercontent.com/u/5047891?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandrudima","html_url":"https://github.com/alexandrudima","followers_url":"https://api.github.com/users/alexandrudima/followers","following_url":"https://api.github.com/users/alexandrudima/following{/other_user}","gists_url":"https://api.github.com/users/alexandrudima/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandrudima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandrudima/subscriptions","organizations_url":"https://api.github.com/users/alexandrudima/orgs","repos_url":"https://api.github.com/users/alexandrudima/repos","events_url":"https://api.github.com/users/alexandrudima/events{/privacy}","received_events_url":"https://api.github.com/users/alexandrudima/received_events","type":"User","site_admin":false},"created_at":"2016-10-22T13:00:18Z","updated_at":"2016-10-22T13:00:18Z","body":"Given more thought, I prefer to leave the current behaviour in, the last `endIndex` is `lineLength + 1`."}}
{"actor":{"display_login":"alexandrudima"}}