{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/telemetry-tools/issues/4","repository_url":"https://api.github.com/repos/mozilla/telemetry-tools","labels_url":"https://api.github.com/repos/mozilla/telemetry-tools/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/telemetry-tools/issues/4/comments","events_url":"https://api.github.com/repos/mozilla/telemetry-tools/issues/4/events","html_url":"https://github.com/mozilla/telemetry-tools/pull/4","id":144676138,"number":4,"title":"Fix Heka message parser backtracking","user":{"login":"Uberi","id":437196,"avatar_url":"https://avatars.githubusercontent.com/u/437196?v=3","gravatar_id":"","url":"https://api.github.com/users/Uberi","html_url":"https://github.com/Uberi","followers_url":"https://api.github.com/users/Uberi/followers","following_url":"https://api.github.com/users/Uberi/following{/other_user}","gists_url":"https://api.github.com/users/Uberi/gists{/gist_id}","starred_url":"https://api.github.com/users/Uberi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Uberi/subscriptions","organizations_url":"https://api.github.com/users/Uberi/orgs","repos_url":"https://api.github.com/users/Uberi/repos","events_url":"https://api.github.com/users/Uberi/events{/privacy}","received_events_url":"https://api.github.com/users/Uberi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-03-30T18:10:44Z","updated_at":"2016-04-01T17:04:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mozilla/telemetry-tools/pulls/4","html_url":"https://github.com/mozilla/telemetry-tools/pull/4","diff_url":"https://github.com/mozilla/telemetry-tools/pull/4.diff","patch_url":"https://github.com/mozilla/telemetry-tools/pull/4.patch"},"body":"* Heka message parsing failed due to an off-by-one error in `BacktrackableFile` - it backtracked one byte too far, which means none of the backtracked tries would ever work. This is trivially fixed.\r\n* Error checking in non-strict mode was too lenient, and caused us to attempt to parse messages when we should have continued backtracking. We now check both the record separator and the unit separator.\r\n    * Note that this also doesn't catch 100% of the cases, but it is off by less than 0.1%.\r\n* Adding `BacktrackableFile.tell()`, which is really useful for debugging parsing errors like this in the future.\r\n\r\nRef: https://bugzilla.mozilla.org/show_bug.cgi?id=1248845\r\n\r\nHere's a [notebook showing the debugging and testing process](https://gist.github.com/Uberi/a3a92bb011c7f3b0e8dc677c91471a10). The results seem to match up with expectations."},"comment":{"url":"https://api.github.com/repos/mozilla/telemetry-tools/issues/comments/204472878","html_url":"https://github.com/mozilla/telemetry-tools/pull/4#issuecomment-204472878","issue_url":"https://api.github.com/repos/mozilla/telemetry-tools/issues/4","id":204472878,"user":{"login":"Uberi","id":437196,"avatar_url":"https://avatars.githubusercontent.com/u/437196?v=3","gravatar_id":"","url":"https://api.github.com/users/Uberi","html_url":"https://github.com/Uberi","followers_url":"https://api.github.com/users/Uberi/followers","following_url":"https://api.github.com/users/Uberi/following{/other_user}","gists_url":"https://api.github.com/users/Uberi/gists{/gist_id}","starred_url":"https://api.github.com/users/Uberi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Uberi/subscriptions","organizations_url":"https://api.github.com/users/Uberi/orgs","repos_url":"https://api.github.com/users/Uberi/repos","events_url":"https://api.github.com/users/Uberi/events{/privacy}","received_events_url":"https://api.github.com/users/Uberi/received_events","type":"User","site_admin":false},"created_at":"2016-04-01T17:04:55Z","updated_at":"2016-04-01T17:04:55Z","body":"I ran the hello world notebook after installing the new version - everything seems to look normal, and the `get_records` run works properly now. There are slight differences in counts, but that's to be expected due to imperfect error correction."}}