{"action":"created","issue":{"url":"https://api.github.com/repos/datalust/seq-import/issues/7","repository_url":"https://api.github.com/repos/datalust/seq-import","labels_url":"https://api.github.com/repos/datalust/seq-import/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/datalust/seq-import/issues/7/comments","events_url":"https://api.github.com/repos/datalust/seq-import/issues/7/events","html_url":"https://github.com/datalust/seq-import/pull/7","id":194807575,"number":7,"title":"Add support for reading events streamed and in compact format","user":{"login":"Suchiman","id":7110884,"avatar_url":"https://avatars.githubusercontent.com/u/7110884?v=3","gravatar_id":"","url":"https://api.github.com/users/Suchiman","html_url":"https://github.com/Suchiman","followers_url":"https://api.github.com/users/Suchiman/followers","following_url":"https://api.github.com/users/Suchiman/following{/other_user}","gists_url":"https://api.github.com/users/Suchiman/gists{/gist_id}","starred_url":"https://api.github.com/users/Suchiman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Suchiman/subscriptions","organizations_url":"https://api.github.com/users/Suchiman/orgs","repos_url":"https://api.github.com/users/Suchiman/repos","events_url":"https://api.github.com/users/Suchiman/events{/privacy}","received_events_url":"https://api.github.com/users/Suchiman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-12-11T02:16:25Z","updated_at":"2016-12-13T01:25:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/datalust/seq-import/pulls/7","html_url":"https://github.com/datalust/seq-import/pull/7","diff_url":"https://github.com/datalust/seq-import/pull/7.diff","patch_url":"https://github.com/datalust/seq-import/pull/7.patch"},"body":"Hi Nicholas, made some improvements, hope you don't mind 😄\r\n\r\n- Allows to specify wether to read input as default or compact json and wether to post it as default or compact json to seq (converting on the fly; doesn't deal with `Renderings` though but they are preserved when passing through, e.g. `--compact-input` and `--compact-output` or default input and default output). \r\n- Events are now read streamed into the buffer allowing to deal with arbitrary large json files. They are temporarily buffered though for the pseudo binary search when the bulk upload failed. I also have another version of the `StreamingEventReader` where the `JsonTextReader` is used in `SupportMultipleContent` mode which yields about 5% better performance but doesn't play nicely with malformed JSON. Anyhow this should fix #4 \r\n- Reduces GC pressure by reusing buffers. Lots of code were allocating objects like MemoryStreams and throwing them away on every iteration. Keeping them around and using `SetLength(0)` allows reusing them so the backing array doesn't need to be re-allocated.\r\n- Added some tests for event conversion and property enrichment\r\n\r\nIn my tests i could import a 160MB compact json file with around 5% cpu consumption and 14~16MB RAM usage, mainly bottlenecked by Seq."},"comment":{"url":"https://api.github.com/repos/datalust/seq-import/issues/comments/266607684","html_url":"https://github.com/datalust/seq-import/pull/7#issuecomment-266607684","issue_url":"https://api.github.com/repos/datalust/seq-import/issues/7","id":266607684,"user":{"login":"Suchiman","id":7110884,"avatar_url":"https://avatars.githubusercontent.com/u/7110884?v=3","gravatar_id":"","url":"https://api.github.com/users/Suchiman","html_url":"https://github.com/Suchiman","followers_url":"https://api.github.com/users/Suchiman/followers","following_url":"https://api.github.com/users/Suchiman/following{/other_user}","gists_url":"https://api.github.com/users/Suchiman/gists{/gist_id}","starred_url":"https://api.github.com/users/Suchiman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Suchiman/subscriptions","organizations_url":"https://api.github.com/users/Suchiman/orgs","repos_url":"https://api.github.com/users/Suchiman/repos","events_url":"https://api.github.com/users/Suchiman/events{/privacy}","received_events_url":"https://api.github.com/users/Suchiman/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T01:25:54Z","updated_at":"2016-12-13T01:25:54Z","body":"I've addressed the easy stuff.\r\n\r\nI've also tested some things around and it looks like i can cut the overhead (measured by commenting the actual POST) in my testcase (compact; 500k events; 80MB) from 3.7s to around ~2.5s by using JsonReader and JsonWriter directly. 50% of that time is still spent in Newtonsoft after that optimization though."}}
{"actor":{"display_login":"Suchiman"}}