{"action":"created","issue":{"url":"https://api.github.com/repos/JDASoftwareGroup/kartothek/issues/31","repository_url":"https://api.github.com/repos/JDASoftwareGroup/kartothek","labels_url":"https://api.github.com/repos/JDASoftwareGroup/kartothek/issues/31/labels{/name}","comments_url":"https://api.github.com/repos/JDASoftwareGroup/kartothek/issues/31/comments","events_url":"https://api.github.com/repos/JDASoftwareGroup/kartothek/issues/31/events","html_url":"https://github.com/JDASoftwareGroup/kartothek/pull/31","id":445472351,"node_id":"MDExOlB1bGxSZXF1ZXN0Mjc5OTE2NDU1","number":31,"title":"Implement efficient partial index serialization","user":{"login":"lr4d","id":44430780,"node_id":"MDQ6VXNlcjQ0NDMwNzgw","avatar_url":"https://avatars1.githubusercontent.com/u/44430780?v=4","gravatar_id":"","url":"https://api.github.com/users/lr4d","html_url":"https://github.com/lr4d","followers_url":"https://api.github.com/users/lr4d/followers","following_url":"https://api.github.com/users/lr4d/following{/other_user}","gists_url":"https://api.github.com/users/lr4d/gists{/gist_id}","starred_url":"https://api.github.com/users/lr4d/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lr4d/subscriptions","organizations_url":"https://api.github.com/users/lr4d/orgs","repos_url":"https://api.github.com/users/lr4d/repos","events_url":"https://api.github.com/users/lr4d/events{/privacy}","received_events_url":"https://api.github.com/users/lr4d/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2019-05-17T14:32:48Z","updated_at":"2019-05-20T09:54:44Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/JDASoftwareGroup/kartothek/pulls/31","html_url":"https://github.com/JDASoftwareGroup/kartothek/pull/31","diff_url":"https://github.com/JDASoftwareGroup/kartothek/pull/31.diff","patch_url":"https://github.com/JDASoftwareGroup/kartothek/pull/31.patch"},"body":"Command used to run benchmarks: `asv run --python=same --show-stderr --bench=Index.track_mem_serialized`\r\n\r\nDiff of benchmark results (`diff old.txt new.txt`) :\r\n```\r\n< [100.00%] ··· =============== =================== =================================== ==========\r\n<                number_values   number_partitions                 dtype\r\n<               --------------- ------------------- ----------------------------------- ----------\r\n<                      10                10           (<class 'int'>, DataType(int64))     1145\r\n<                      10                10          (<class 'str'>, DataType(string))     1206\r\n<                      10               1000          (<class 'int'>, DataType(int64))    128558\r\n<                      10               1000         (<class 'str'>, DataType(string))    128649\r\n<                     1000               10           (<class 'int'>, DataType(int64))    120304\r\n<                     1000               10          (<class 'str'>, DataType(string))    130451\r\n<                     1000              1000          (<class 'int'>, DataType(int64))   12900304\r\n<                     1000              1000         (<class 'str'>, DataType(string))   12910451\r\n<               =============== =================== =================================== ==========\r\n---\r\n> [100.00%] ··· =============== =================== =================================== ========\r\n>                number_values   number_partitions                 dtype\r\n>               --------------- ------------------- ----------------------------------- --------\r\n>                      10                10           (<class 'int'>, DataType(int64))    792\r\n>                      10                10          (<class 'str'>, DataType(string))    707\r\n>                      10               1000          (<class 'int'>, DataType(int64))    6681\r\n>                      10               1000         (<class 'str'>, DataType(string))    6596\r\n>                     1000               10           (<class 'int'>, DataType(int64))    6349\r\n>                     1000               10          (<class 'str'>, DataType(string))    6272\r\n>                     1000              1000          (<class 'int'>, DataType(int64))   109994\r\n>                     1000              1000         (<class 'str'>, DataType(string))   109917\r\n>               =============== =================== =================================== ========\r\n```"},"comment":{"url":"https://api.github.com/repos/JDASoftwareGroup/kartothek/issues/comments/493918074","html_url":"https://github.com/JDASoftwareGroup/kartothek/pull/31#issuecomment-493918074","issue_url":"https://api.github.com/repos/JDASoftwareGroup/kartothek/issues/31","id":493918074,"node_id":"MDEyOklzc3VlQ29tbWVudDQ5MzkxODA3NA==","user":{"login":"lr4d","id":44430780,"node_id":"MDQ6VXNlcjQ0NDMwNzgw","avatar_url":"https://avatars1.githubusercontent.com/u/44430780?v=4","gravatar_id":"","url":"https://api.github.com/users/lr4d","html_url":"https://github.com/lr4d","followers_url":"https://api.github.com/users/lr4d/followers","following_url":"https://api.github.com/users/lr4d/following{/other_user}","gists_url":"https://api.github.com/users/lr4d/gists{/gist_id}","starred_url":"https://api.github.com/users/lr4d/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lr4d/subscriptions","organizations_url":"https://api.github.com/users/lr4d/orgs","repos_url":"https://api.github.com/users/lr4d/repos","events_url":"https://api.github.com/users/lr4d/events{/privacy}","received_events_url":"https://api.github.com/users/lr4d/received_events","type":"User","site_admin":false},"created_at":"2019-05-20T09:54:44Z","updated_at":"2019-05-20T09:54:44Z","author_association":"CONTRIBUTOR","body":"Benchmarks on `--bench=Index`, `with pyarrow=0.12.1`:\r\n\r\n``` diff\r\ndiff json-2019-05-20.txt parquet-2019-05-20.txt\r\n14,17c14,17\r\n<                  -1      1000    14.5±0.5ms    346±10ms\r\n<                  -1     10000     191±4ms     4.20±0.06s\r\n<                  1       1000    685±100μs    7.34±0.3ms\r\n<                  1      10000     665±40μs    6.81±0.2ms\r\n---\r\n>                  -1      1000    15.1±0.9ms    411±60ms\r\n>                  -1     10000     201±10ms    4.23±0.03s\r\n>                  1       1000     662±20μs    7.32±0.3ms\r\n>                  1      10000     692±90μs    6.93±0.4ms\r\n21,30c21,30\r\n< [ 64.29%] ··· ======== ======== ============ =============\r\n<               --                          param3\r\n<               ----------------- --------------------------\r\n<                param1   param2       10           100\r\n<               ======== ======== ============ =============\r\n<                  -1      1000    5.17±0.5ms    5.31±0.3ms\r\n<                  -1     10000    31.7±0.7ms     31.8±1ms\r\n<                  1       1000    2.37±0.2ms    2.43±0.2ms\r\n<                  1      10000    1.83±0.1ms   1.71±0.03ms\r\n<               ======== ======== ============ =============\r\n---\r\n> [ 64.29%] ··· ======== ======== ============ ============\r\n>               --                          param3\r\n>               ----------------- -------------------------\r\n>                param1   param2       10          100\r\n>               ======== ======== ============ ============\r\n>                  -1      1000    5.58±0.4ms   5.76±0.3ms\r\n>                  -1     10000     34.2±2ms    34.7±0.3ms\r\n>                  1       1000    2.73±0.7ms   2.56±0.4ms\r\n>                  1      10000    1.99±0.1ms   2.16±0.5ms\r\n>               ======== ======== ============ ============\r\n33,44c33,44\r\n< [ 71.43%] ··· =============== =================== =================================== =============\r\n<                number_values   number_partitions                 dtype\r\n<               --------------- ------------------- ----------------------------------- -------------\r\n<                      10                10           (<class 'int'>, DataType(int64))   3.15±0.06ms\r\n<                      10                10          (<class 'str'>, DataType(string))   3.20±0.08ms\r\n<                      10               1000          (<class 'int'>, DataType(int64))   4.66±0.08ms\r\n<                      10               1000         (<class 'str'>, DataType(string))   5.05±0.06ms\r\n<                     1000               10           (<class 'int'>, DataType(int64))    5.58±0.1ms\r\n<                     1000               10          (<class 'str'>, DataType(string))   6.17±0.08ms\r\n<                     1000              1000          (<class 'int'>, DataType(int64))     176±2ms\r\n<                     1000              1000         (<class 'str'>, DataType(string))     232±2ms\r\n<               =============== =================== =================================== =============\r\n---\r\n> [ 71.43%] ··· =============== =================== =================================== ============\r\n>                number_values   number_partitions                 dtype\r\n>               --------------- ------------------- ----------------------------------- ------------\r\n>                      10                10           (<class 'int'>, DataType(int64))    3.37±1ms\r\n>                      10                10          (<class 'str'>, DataType(string))   3.36±0.3ms\r\n>                      10               1000          (<class 'int'>, DataType(int64))   4.84±0.1ms\r\n>                      10               1000         (<class 'str'>, DataType(string))   5.12±0.1ms\r\n>                     1000               10           (<class 'int'>, DataType(int64))   5.57±0.1ms\r\n>                     1000               10          (<class 'str'>, DataType(string))   6.31±0.2ms\r\n>                     1000              1000          (<class 'int'>, DataType(int64))    177±6ms\r\n>                     1000              1000         (<class 'str'>, DataType(string))    235±7ms\r\n>               =============== =================== =================================== ============\r\n47,58c47,58\r\n< [ 78.57%] ··· =============== =================== =================================== =============\r\n<                number_values   number_partitions                 dtype\r\n<               --------------- ------------------- ----------------------------------- -------------\r\n<                      10                10           (<class 'int'>, DataType(int64))   3.21±0.07ms\r\n<                      10                10          (<class 'str'>, DataType(string))    3.44±0.3ms\r\n<                      10               1000          (<class 'int'>, DataType(int64))    4.92±0.7ms\r\n<                      10               1000         (<class 'str'>, DataType(string))   5.18±0.08ms\r\n<                     1000               10           (<class 'int'>, DataType(int64))    6.00±0.1ms\r\n<                     1000               10          (<class 'str'>, DataType(string))    6.37±0.2ms\r\n<                     1000              1000          (<class 'int'>, DataType(int64))     190±1ms\r\n<                     1000              1000         (<class 'str'>, DataType(string))     229±3ms\r\n<               =============== =================== =================================== =============\r\n---\r\n> [ 78.57%] ··· =============== =================== =================================== ============\r\n>                number_values   number_partitions                 dtype\r\n>               --------------- ------------------- ----------------------------------- ------------\r\n>                      10                10           (<class 'int'>, DataType(int64))    4.16±1ms\r\n>                      10                10          (<class 'str'>, DataType(string))   3.29±0.3ms\r\n>                      10               1000          (<class 'int'>, DataType(int64))   5.02±0.1ms\r\n>                      10               1000         (<class 'str'>, DataType(string))   5.00±0.1ms\r\n>                     1000               10           (<class 'int'>, DataType(int64))   5.83±0.3ms\r\n>                     1000               10          (<class 'str'>, DataType(string))   6.22±0.1ms\r\n>                     1000              1000          (<class 'int'>, DataType(int64))    190±3ms\r\n>                     1000              1000         (<class 'str'>, DataType(string))    237±5ms\r\n>               =============== =================== =================================== ============\r\n64,71c64,71\r\n<                      10                10           (<class 'int'>, DataType(int64))   2.11±0.02ms\r\n<                      10                10          (<class 'str'>, DataType(string))   2.06±0.03ms\r\n<                      10               1000          (<class 'int'>, DataType(int64))   3.13±0.06ms\r\n<                      10               1000         (<class 'str'>, DataType(string))   3.09±0.08ms\r\n<                     1000               10           (<class 'int'>, DataType(int64))   4.56±0.06ms\r\n<                     1000               10          (<class 'str'>, DataType(string))   4.51±0.03ms\r\n<                     1000              1000          (<class 'int'>, DataType(int64))     120±7ms\r\n<                     1000              1000         (<class 'str'>, DataType(string))     128±6ms\r\n---\r\n>                      10                10           (<class 'int'>, DataType(int64))   2.18±0.02ms\r\n>                      10                10          (<class 'str'>, DataType(string))   2.05±0.03ms\r\n>                      10               1000          (<class 'int'>, DataType(int64))   3.27±0.06ms\r\n>                      10               1000         (<class 'str'>, DataType(string))   3.09±0.05ms\r\n>                     1000               10           (<class 'int'>, DataType(int64))   4.65±0.06ms\r\n>                     1000               10          (<class 'str'>, DataType(string))   4.47±0.03ms\r\n>                     1000              1000          (<class 'int'>, DataType(int64))     135±6ms\r\n>                     1000              1000         (<class 'str'>, DataType(string))     131±10ms\r\n78,83c78,83\r\n<                      10                10           (<class 'int'>, DataType(int64))   2.52±0.09μs\r\n<                      10                10          (<class 'str'>, DataType(string))   1.71±0.04μs\r\n<                      10               1000          (<class 'int'>, DataType(int64))   2.47±0.06μs\r\n<                      10               1000         (<class 'str'>, DataType(string))   1.74±0.05μs\r\n<                     1000               10           (<class 'int'>, DataType(int64))    2.57±0.2μs\r\n<                     1000               10          (<class 'str'>, DataType(string))    1.79±0.1μs\r\n---\r\n>                      10                10           (<class 'int'>, DataType(int64))   2.55±0.08μs\r\n>                      10                10          (<class 'str'>, DataType(string))   1.86±0.09μs\r\n>                      10               1000          (<class 'int'>, DataType(int64))    2.42±0.1μs\r\n>                      10               1000         (<class 'str'>, DataType(string))   1.72±0.06μs\r\n>                     1000               10           (<class 'int'>, DataType(int64))    2.89±0.2μs\r\n>                     1000               10          (<class 'str'>, DataType(string))     3.29±2μs\r\n85c85\r\n<                     1000              1000         (<class 'str'>, DataType(string))   1.67±0.03μs\r\n---\r\n>                     1000              1000         (<class 'str'>, DataType(string))   1.75±0.07μs\r\n89,100c89,100\r\n< [100.00%] ··· =============== =================== =================================== ==========\r\n<                number_values   number_partitions                 dtype\r\n<               --------------- ------------------- ----------------------------------- ----------\r\n<                      10                10           (<class 'int'>, DataType(int64))     1145\r\n<                      10                10          (<class 'str'>, DataType(string))     1206\r\n<                      10               1000          (<class 'int'>, DataType(int64))    128558\r\n<                      10               1000         (<class 'str'>, DataType(string))    128649\r\n<                     1000               10           (<class 'int'>, DataType(int64))    120304\r\n<                     1000               10          (<class 'str'>, DataType(string))    130451\r\n<                     1000              1000          (<class 'int'>, DataType(int64))   12900304\r\n<                     1000              1000         (<class 'str'>, DataType(string))   12910451\r\n<               =============== =================== =================================== ==========\r\n---\r\n> [100.00%] ··· =============== =================== =================================== ========\r\n>                number_values   number_partitions                 dtype\r\n>               --------------- ------------------- ----------------------------------- --------\r\n>                      10                10           (<class 'int'>, DataType(int64))    794\r\n>                      10                10          (<class 'str'>, DataType(string))    709\r\n>                      10               1000          (<class 'int'>, DataType(int64))    6683\r\n>                      10               1000         (<class 'str'>, DataType(string))    6598\r\n>                     1000               10           (<class 'int'>, DataType(int64))    6351\r\n>                     1000               10          (<class 'str'>, DataType(string))    6274\r\n>                     1000              1000          (<class 'int'>, DataType(int64))   109996\r\n>                     1000              1000         (<class 'str'>, DataType(string))   109919\r\n>               =============== =================== =================================== ========\r\n```"}}
{"actor":{"display_login":"lr4d"}}