{"push_id":1440071826,"size":7,"distinct_size":2,"ref":"refs/heads/SNAP-1015-new","head":"e278636f028a07cc598215ab4fd40a30d0272426","before":"841668598370b44ad489d15728368cdcd04994dc","commits":[{"sha":"aa012dd5fe2cf87b6e1b7b7100ff1f8a4725be2c","author":{"name":"Sumedh Wale","email":"5d0c5259ad7ea5b78193c34716d4a13afa4c0d21@snappydata.io"},"message":"explicitly set batchSize in CachedBatchScanDUnitTest to fix failure","distinct":false,"url":"https://api.github.com/repos/SnappyDataInc/snappydata/commits/aa012dd5fe2cf87b6e1b7b7100ff1f8a4725be2c"},{"sha":"788b88c1c9d633fcd96d244dd680f35b4b77c9ca","author":{"name":"Rishitesh Mishra","email":"4d130cb319c3dc95dbe267e244c0dfe82a55ed42@snappydata.io"},"message":"Snappy job server app APIs are still taking SnappyContext as in input… (#419)\n\n* Snappy job server app APIs are still taking SnappyContext as in input parameter. Changed it to SnappySession.\r\n\r\n* Test changes\r\n\r\n* Merged and changed Jobs API for SnappySession as callback arg.\r\n\r\n* Renamed SnappyContextFactory to SnappySessionFactory.\r\n\r\n* Changed the test to check default connection pool to be Tomcat and not Hikari.","distinct":false,"url":"https://api.github.com/repos/SnappyDataInc/snappydata/commits/788b88c1c9d633fcd96d244dd680f35b4b77c9ca"},{"sha":"9aa636be282697de5c81ec751a4ffd6a662c14b9","author":{"name":"Sumedh Wale","email":"5d0c5259ad7ea5b78193c34716d4a13afa4c0d21@snappydata.io"},"message":"actually fixing the failures in CachedBatchScanDUnitTest by allowing flush with batch size smaller than min batch size (currently 200)","distinct":false,"url":"https://api.github.com/repos/SnappyDataInc/snappydata/commits/9aa636be282697de5c81ec751a4ffd6a662c14b9"},{"sha":"dfc016db9a9de2b16af7a2474c710a6c00a3a38f","author":{"name":"Sumedh Wale","email":"5d0c5259ad7ea5b78193c34716d4a13afa4c0d21@snappydata.io"},"message":"[SNAP-1191] Basic plan caching (without constant tokenization)\n\nAdd plan caching and reuse of SparkPlan, RDD and PlanInfo.\nCurrently it is exact string match and tokenization support for constants\nembedded in query string will be added later.\n\n- Added a new CachedDataFrame which is created by CacheLoader on a cache miss.\n  It provides a generic \"collectWithHandler\" method that allows collecting results\n  using a custom handler (which can, for example, put into local BlockManager like done by SparkSQLExecuteImpl)\n- Compress partition results like that done in SparkPlan.getByteArrayRdd in static\n  CachedDataFrame.apply() and corresponding decodeUnsafeRows method to decode\n- Pass through the above compressed byte result end-to-end from executor to driver to DRDA server.\n  The server invokes decodeUnsafeRows avoiding any additional deserialization or other cost for the additional transition.\n- For the case of \"take\" i.e. limit, the executeTake uses similar approach as by SparkPlan.executeTake.\n  However, instead of decoding every row it will try to get in full compressed blocks as much\n  as possible and decode only the remaining rows in last block (CachedDataFrame.takeRows).\n  Thus collectWithHandler needs two handlers -- one for the partition-wise result, and other for decoding if required.\n- The localBlockStoreResultHandler and localBlockStoreDecoder are the handlers used by SparkSQLExecuteImpl.\n  So only for limit queries will there be any decoding of the residual rows on lead.\n- Added support for sending number of processors as part of GemXD profile (number of executors is\n  different and by default 2 times the processor count) and use that to set the shuffle partitions\n  in SnappyConf. This reduces the default shuffle size to be only as much as number of processors.\n- Send bucket set as bit set so that lookup is faster (an array lookup instead of a hash set).\n- JSON conversion support for complex types in JDBC queries. A query hint (complexTypeAsJson) is required.\n- add handling of fixed CHAR types in SparkSQLExecuteImpl\n  - for CHAR types now blank padding as per the precision or truncating if size is larger\n  - fixes failure in StringAsVarcharDUnitTest\n- support StartsWith in MIN/MAX support for column batches; added test for LIKE 'XYZ%' kind of\n  queries in CachedBatchScanDUnitTest\n- fixed the case where no primary may be present on an executor\n- fix for a couple of issues with dictionary optimization\n  - Added a \"canConsume\" method to BatchConsumer trait which is implemented by checking for\n    matching schema (i.e. the output should be a subset of ColumnTableScan output) in\n    LocalJoin and SnappyHashAggregateExec that implement BatchConsumer; also re-arrange\n    the incoming input variables of ColumnTableScan in case its output order does\n    not match BatchConsumer's expected input order (e.g. a projection in between)\n  - Handle the case of null keys in group by and local join with dictionary optimization.\n    Now expand the dictionary to have null values at the last index (to avoid additional\n        checks throughout the generated code) and the dictionaryIndex variable will be\n    initialized to the last index. Use a separate dictionaryLength variable and\n    also re-factored the common dictionary array initialization code into ObjectHashMapAccessor\n- added sqlUncached method to SnappySession and SnappyContext that with skip all caching\n- adjusting expected partitions with no RDD delinking in NorthWind tests\n- updated ColumnCacheBenchmark to use sparkSession for Spark runs\n- remove optimized=F case in TPCETrade test; no longer useful in comparison and does not\n  work anymore for LocalJoin","distinct":false,"url":"https://api.github.com/repos/SnappyDataInc/snappydata/commits/dfc016db9a9de2b16af7a2474c710a6c00a3a38f"},{"sha":"a8bc057eeb17d3cb726ba6a9f50bf8020bed995a","author":{"name":"Sumedh Wale","email":"5d0c5259ad7ea5b78193c34716d4a13afa4c0d21@snappydata.io"},"message":"[SNAP-1193] Optimized Collect aggregate plan to avoid last step exchange (#445)\n\n- New CollectAggregateExec plan that will take a last stage SnappyHashAggregateExec and\r\n  execute it on driver. Right now only for non-groupBy/distinct cases due to concerns\r\n  with potentially large evaluation but this must be revisited once Spark adds plan statistics.\r\n- This lets code-generation happen as in SnappyHashAggregateExec and just switches the producer\r\n  of rows to be the InputAdapter around the result of runJob.\r\n  Saving is of the order of 20ms for simple count etc.\r\n- Adding a common SnappySession for SnappyTableStatsProviderService.\r\n  Creating a new SnappySession in every run of SnappyTableStatsProviderService is somewhat costly\r\n  due to initialization of hive catalog everytime so creating a shared one with cleanup in stop.","distinct":false,"url":"https://api.github.com/repos/SnappyDataInc/snappydata/commits/a8bc057eeb17d3cb726ba6a9f50bf8020bed995a"},{"sha":"18b896c2f1da5d2bf755f52cc8d805c88bca44e5","author":{"name":"sonal","email":"923ecad8d6182a636bfe80ce9b010244edffb543@snappydata.io"},"message":"Merge branch 'master' into SNAP-1015-new","distinct":true,"url":"https://api.github.com/repos/SnappyDataInc/snappydata/commits/18b896c2f1da5d2bf755f52cc8d805c88bca44e5"},{"sha":"e278636f028a07cc598215ab4fd40a30d0272426","author":{"name":"sonal","email":"923ecad8d6182a636bfe80ce9b010244edffb543@snappydata.io"},"message":"- Changing signature as per new changes in master","distinct":true,"url":"https://api.github.com/repos/SnappyDataInc/snappydata/commits/e278636f028a07cc598215ab4fd40a30d0272426"}]}
{"actor":{"display_login":"sonalsagarwal"}}