{"push_id":2769046430,"size":14,"distinct_size":14,"ref":"refs/heads/data-size-aggregations","head":"47e208122d230b57dcdf168fcea61f0fb66e84dc","before":"b48e185080dda5ad4c68b60141e6ab183858c187","commits":[{"sha":"5315bc5e6be3544c55834475a980aadff3debae6","author":{"name":"Martin Traverso","email":"a3f23a76a6203525fe5c769a7a3ec65aa430adc3@gmail.com"},"message":"Add SpookyHashV2 hash functions","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/5315bc5e6be3544c55834475a980aadff3debae6"},{"sha":"6dcd67fdffbe291ca68f1c5fe8936a575cce86df","author":{"name":"Ying Su","email":"bdbe458668c4e46cd6ebc186bb28a68ef7fe6ad7@fb.com"},"message":"Inline only simple expressions in predicate pushdown\n\nThis is to fix github issue #10455.\n\nWe used to inline all predicate when doing push down predicates. This could have problems when the projection columns contains complex expressions like the following:\n\nWITH\nt1 (v) AS (VALUES 1),\nt2 AS( select if(v = 0, v, v) v from t1 ),\nt3 AS( select if(v = 0, v, v) v from t2 ),\nt4 AS( select if(v = 0, v, v) v from t3 ),\nt5 AS( select if(v = 0, v, v) v from t4 ),\nt6 AS( select if(v = 0, v, v) v from t5 ),\nt7 AS( select if(v = 0, v, v) v from t6 ),\nt8 AS( select if(v = 0, v, v) v from t7 ),\nt9 AS( select if(v = 0, v, v) v from t8 ),\nt10 AS( select if(v = 0, v, v) v from t9 ),\nt11 AS( select if(v = 0, v, v) v from t10 ),\nt12 AS( select if(v = 0, v, v) v from t11 ),\nt13 AS( select if(v = 0, v, v) v from t12 ),\nt14 AS( select if(v = 0, v, v) v from t13 ),\nt15 AS( select if(v = 0, v, v) v from t14 ),\nt16 AS( select if(v = 0, v, v) v from t15 )\nselect *\nfrom t16\nwhere v = 0\n\nThis short-term fix is to adjust the inlining heuristics to only do it if the expressions are trivial or appear only once (similar to how the InlineProjections rule works)","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/6dcd67fdffbe291ca68f1c5fe8936a575cce86df"},{"sha":"f93d5e5c1d12b5380927f6f6541a8b068ccbe49b","author":{"name":"David Phillips","email":"aa743a0aaec8f7d7a1f01442503957f4d7a2d634@acz.org"},"message":"Update to Airbase 82\n\nThis version guarantees the JVM will terminate on OOM during tests,\nand fixes runnning Surefire tests on later JDKs.","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/f93d5e5c1d12b5380927f6f6541a8b068ccbe49b"},{"sha":"a2c6c129fd2b1dc2e09c21b56c2cf481b22f809c","author":{"name":"Haozhun Jin","email":"8baac4a5cf68bd25ac08ee741592ae7b45274707@gmail.com"},"message":"Skip currentConstraint field when serializing TableScanNode\n\ncurrentConstraint can be pretty complex. As a result, it may incur\na significant cost to serialize, to hold in memory as a JSON,\nand to transport over network.","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/a2c6c129fd2b1dc2e09c21b56c2cf481b22f809c"},{"sha":"7b33a4674782032691c90ab416812a77b93491ae","author":{"name":"Haozhun Jin","email":"8baac4a5cf68bd25ac08ee741592ae7b45274707@gmail.com"},"message":"Make PlanFragmenter.createSubPlans not static","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/7b33a4674782032691c90ab416812a77b93491ae"},{"sha":"73d94c294ee58d740c3b84b63fe211f9857a2a4d","author":{"name":"Haozhun Jin","email":"8baac4a5cf68bd25ac08ee741592ae7b45274707@gmail.com"},"message":"Reduce LocalQueryRunner constructor overload","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/73d94c294ee58d740c3b84b63fe211f9857a2a4d"},{"sha":"2730762662b99292ee37be99e71f39b731119c76","author":{"name":"Haozhun Jin","email":"8baac4a5cf68bd25ac08ee741592ae7b45274707@gmail.com"},"message":"Limit number of stages in a query","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/2730762662b99292ee37be99e71f39b731119c76"},{"sha":"9899432eaf7d444517e24ad806ba0c6fc33ae433","author":{"name":"Haozhun Jin","email":"8baac4a5cf68bd25ac08ee741592ae7b45274707@gmail.com"},"message":"Add session property to ease rollout of stage count limit","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/9899432eaf7d444517e24ad806ba0c6fc33ae433"},{"sha":"9c45d9a2c6eed4c1cd06f7d375d4bf077a08438c","author":{"name":"Ying Su","email":"bdbe458668c4e46cd6ebc186bb28a68ef7fe6ad7@fb.com"},"message":"Grow the batch size fro OrcRecordReader from 1 row\n\nIn the past we have seen large number of humongous allocations on the\ncluster worker nodes. The size of each block could be as high\nas 80MB, and the total allocation could be as high as 100GB per worker\nnode. This situation happens when reading map or array types. One of the\nreason is that when we read the nested types like map and array, we\ndidn't take the number of elements within each cell into account when\ncaculating the batch size. A single map cell could contain 10K entries\nin some cases and the total data entries read could be 10M. Currently\nwe read the first block and bound the batch size accordingly, but the\nfirst block might already contain too many (e.g. millions of) entries.\n\nThis commit is to change the first read from 1024 rows to 1 row, then\nexponentially grow the batch size until it hits the 1024-row bound or\nthe max block size bound (16MB), then adjust the batch size for each\nbatch later on. The growth factor BATCH_SIZE_GROWTH_FACTOR is defined\nas 2 for now.\n\nTo show the result of this change, we did an experiment reading a\ncolumn of type MAP<INT,ARRAY<BIGINT>> where each row contains around\n10K entries. Before the change the block sizes were around 80MB, and\nafter this change the block sizes were bounded around 16MB.\n\nThe number of rows read and block sizes before the change:\nReading 979 rows. block size: 76853564\nReading 979 rows. block size: 75622553\nReading 1011 rows. block size: 80844513\nReading 1024 rows. block size: 81683180\nReading 70 rows. block size: 5939036\nReading 1022 rows. block size: 77635114\nReading 997 rows. block size: 75353930\nReading 989 rows. block size: 81052672\n\nThe number of rows read and block sizes after the change:\n(suppose the growth factor is 32)\nReading one column:\nReading 1 rows. block size: 81185\nReading 32 rows. block size: 2560714\nReading 206 rows. block size: 16128454\nReading 206 rows. block size: 16150072\nReading 206 rows. block size: 16120201\nReading 206 rows. block size: 15966427\nReading 145 rows. block size: 11342435\n\nReading two columns:\nReading 1 rows. block size 80141 block size 64409\nReading 32 rows. block size 2423968 block size 2821975\nReading 99 rows. block size 7631865 block size 7972065\nReading 99 rows. block size 7481655 block size 8050815\n\n(suppose the growth factor is 2)\nReading 1 rows Block size 78956\nReading 2 rows Block size 110110\nReading 4 rows Block size 250065\nReading 8 rows Block size 424828\nReading 16 rows Block size 949918\nReading 32 rows Block size 2493616\nReading 64 rows Block size 5577162\nReading 128 rows Block size 12501611\nReading 190 rows Block size 18751822\nReading 190 rows Block size 18767870\nReading 190 rows Block size 18718009\nReading 170 rows Block size 12666075","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/9c45d9a2c6eed4c1cd06f7d375d4bf077a08438c"},{"sha":"50ed39f50ffe60fb03cd9e8f467157ce2db81638","author":{"name":"Wenlei Xie","email":"3b0cca290106b3a27923ab559ce16239893a2cad@gmail.com"},"message":"Cleanup legacy handling for TRY\n\nThe TRY expression used to be handled in a special way similar to lambda\nexpressions. #8889 removed this special handling by making the TRY expression\na syntactic sugar using lambdas.\n\nThis commit cleans up the leftover name `LambdaAndTryExpression` and\nremoves the unused `tryMethodMap`.","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/50ed39f50ffe60fb03cd9e8f467157ce2db81638"},{"sha":"175ad301ad86b5b5f5eae58f0fd97aecc0345014","author":{"name":"Andrii Rosa","email":"0dfa5100905c475f6ca556aa6238bcaabaf1f5ed@fb.com"},"message":"Fix SingleRowBlock#getSizeInBytes\n\ngetSizeInBytes should return the size of a single entry","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/175ad301ad86b5b5f5eae58f0fd97aecc0345014"},{"sha":"2f90d7a3166b820ab8f60258505da395bc67b05f","author":{"name":"Andrii Rosa","email":"0dfa5100905c475f6ca556aa6238bcaabaf1f5ed@fb.com"},"message":"Support hidden aggregation functions","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/2f90d7a3166b820ab8f60258505da395bc67b05f"},{"sha":"4cc73a1931b01ef77cf42cd7d33ed53fd35de2a6","author":{"name":"Andrii Rosa","email":"0dfa5100905c475f6ca556aa6238bcaabaf1f5ed@fb.com"},"message":"Implement Block#getEstimatedDataSizeForStats","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/4cc73a1931b01ef77cf42cd7d33ed53fd35de2a6"},{"sha":"47e208122d230b57dcdf168fcea61f0fb66e84dc","author":{"name":"Andrii Rosa","email":"0dfa5100905c475f6ca556aa6238bcaabaf1f5ed@fb.com"},"message":"Add internal aggregations over estimated data size for stats\n\nAdd $internal$sum_data_size_for_stats and $internal$max_data_size_for_stats.\n\nThese aggregation functions will be used to computed the total data size and the\nmax value size statistics for VARCHAR/CHAR/VARBINARY columns and for complex\ntypes columns.","distinct":true,"url":"https://api.github.com/repos/arhimondr/presto/commits/47e208122d230b57dcdf168fcea61f0fb66e84dc"}]}
{"actor":{"display_login":"arhimondr"}}