{"action":"created","issue":{"url":"https://api.github.com/repos/pieroxy/lz-string/issues/120","repository_url":"https://api.github.com/repos/pieroxy/lz-string","labels_url":"https://api.github.com/repos/pieroxy/lz-string/issues/120/labels{/name}","comments_url":"https://api.github.com/repos/pieroxy/lz-string/issues/120/comments","events_url":"https://api.github.com/repos/pieroxy/lz-string/issues/120/events","html_url":"https://github.com/pieroxy/lz-string/issues/120","id":361967918,"node_id":"MDU6SXNzdWUzNjE5Njc5MTg=","number":120,"title":"Selective node splitting","user":{"login":"JobLeonard","id":259840,"node_id":"MDQ6VXNlcjI1OTg0MA==","avatar_url":"https://avatars3.githubusercontent.com/u/259840?v=4","gravatar_id":"","url":"https://api.github.com/users/JobLeonard","html_url":"https://github.com/JobLeonard","followers_url":"https://api.github.com/users/JobLeonard/followers","following_url":"https://api.github.com/users/JobLeonard/following{/other_user}","gists_url":"https://api.github.com/users/JobLeonard/gists{/gist_id}","starred_url":"https://api.github.com/users/JobLeonard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JobLeonard/subscriptions","organizations_url":"https://api.github.com/users/JobLeonard/orgs","repos_url":"https://api.github.com/users/JobLeonard/repos","events_url":"https://api.github.com/users/JobLeonard/events{/privacy}","received_events_url":"https://api.github.com/users/JobLeonard/received_events","type":"User","site_admin":false},"labels":[{"id":38923761,"node_id":"MDU6TGFiZWwzODkyMzc2MQ==","url":"https://api.github.com/repos/pieroxy/lz-string/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-09-19T23:49:00Z","updated_at":"2018-09-24T22:10:03Z","closed_at":null,"author_association":"COLLABORATOR","body":"So @gloryknight came up with a very interesting and simple heuristic that apparently works really well in many JSON files: whenever a `,` is encountered (can be set to another character, like a space or newline), LZ is forced to stop the current substring, and start anew (with a leading comma), _except_ when the current substring starts with a comma. See these two pull-requests:\r\n\r\nhttps://github.com/JobLeonard/lz-string/pull/3\r\n\r\nhttps://github.com/JobLeonard/lz-string/pull/4\r\n\r\nThe reason for this being effective is a bit subtle: imagine that we have a string we are scanning through, and the next set of characters will be `abcdefg`. Furthermore, our dictionary already has the substrings `abc`, `abcd` and `defg` (plus the necessary substrings to get to this point), but _not_ `efg`. Obviously, the ideal combination of tokens would be `abc` + `defg`. Instead we'll get `abcd` + `e` + `f` + `g`. This can happen quite often in LZ. So how to avoid this? Well, I guess gloryknight's insight was that not all characters are created equal here; they can have special functions. One of those is as *separator characters*. Think of natural language: or words are separated by spaces, so if we split on the space character (and similar separator like newlines, dots, commas) we would converge on identical substrings much quicker.\r\n\r\nSince LZString is most commonly used whn compressing JSON, which strips out all unnecessary whitespace, the `,` is the option that seems to improve compression performance (although maybe `{` and `:` also make sense, or maybe all three?). In his tests it gave significant compression benefits at a small perf cost.\r\n\r\nThe best bit? This is perfectly backwards compatible with previous codes: the output can be decompressed by the same function as before."},"comment":{"url":"https://api.github.com/repos/pieroxy/lz-string/issues/comments/424142259","html_url":"https://github.com/pieroxy/lz-string/issues/120#issuecomment-424142259","issue_url":"https://api.github.com/repos/pieroxy/lz-string/issues/120","id":424142259,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDE0MjI1OQ==","user":{"login":"pieroxy","id":1426854,"node_id":"MDQ6VXNlcjE0MjY4NTQ=","avatar_url":"https://avatars0.githubusercontent.com/u/1426854?v=4","gravatar_id":"","url":"https://api.github.com/users/pieroxy","html_url":"https://github.com/pieroxy","followers_url":"https://api.github.com/users/pieroxy/followers","following_url":"https://api.github.com/users/pieroxy/following{/other_user}","gists_url":"https://api.github.com/users/pieroxy/gists{/gist_id}","starred_url":"https://api.github.com/users/pieroxy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pieroxy/subscriptions","organizations_url":"https://api.github.com/users/pieroxy/orgs","repos_url":"https://api.github.com/users/pieroxy/repos","events_url":"https://api.github.com/users/pieroxy/events{/privacy}","received_events_url":"https://api.github.com/users/pieroxy/received_events","type":"User","site_admin":false},"created_at":"2018-09-24T22:10:03Z","updated_at":"2018-09-24T22:10:03Z","author_association":"OWNER","body":"I agree with you. I'm not saying this is a bad idea at all, what I'm just saying is that  `abc` + `defg` is not *obviously* a better choice than to `abcd` + `e` + `f` + `g`. As you said, it depends.\r\n\r\nAs far as releasing is concerned, I would be a bit more cautious than you here. People have gazillions of data snippets compressed in browsers all over the world, and releasing a version that is unable to decompress it could have big impacts. That said, if all unit tests pass, let's go indeed."}}
{"actor":{"display_login":"pieroxy"}}