{"push_id":975523517,"size":67,"distinct_size":67,"ref":"refs/heads/master","head":"c4d5ad80c8091c961646a82e85ecbc335b8ffe2d","before":"140ddef373680cb08a3948a883b172dc80814170","commits":[{"sha":"edf4a0e62e6fdb849cca4f23a7060da5ec782b07","author":{"email":"251f4707aac0b279827494b1f338ba619a5a5409@gmail.com","name":"Nam Pham"},"message":"[SPARK-12986][DOC] Fix pydoc warnings in mllib/regression.py\n\nI have fixed the warnings by running \"make html\" under \"python/docs/\". They are caused by not having blank lines around indented paragraphs.\n\nAuthor: Nam Pham <phamducnam@gmail.com>\n\nCloses #11025 from nampham2/SPARK-12986.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/edf4a0e62e6fdb849cca4f23a7060da5ec782b07"},{"sha":"06f0df6df204c4722ff8a6bf909abaa32a715c41","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-8964] [SQL] Use Exchange to perform shuffle in Limit\n\nThis patch changes the implementation of the physical `Limit` operator so that it relies on the `Exchange` operator to perform data movement rather than directly using `ShuffledRDD`. In addition to improving efficiency, this lays the necessary groundwork for further optimization of limit, such as limit pushdown or whole-stage codegen.\n\nAt a high-level, this replaces the old physical `Limit` operator with two new operators, `LocalLimit` and `GlobalLimit`. `LocalLimit` performs per-partition limits, while `GlobalLimit` applies the final limit to a single partition; `GlobalLimit`'s declares that its `requiredInputDistribution` is `SinglePartition`, which will cause the planner to use an `Exchange` to perform the appropriate shuffles. Thus, a logical `Limit` appearing in the middle of a query plan will be expanded into `LocalLimit -> Exchange to one partition -> GlobalLimit`.\n\nIn the old code, calling `someDataFrame.limit(100).collect()` or `someDataFrame.take(100)` would actually skip the shuffle and use a fast-path which used `executeTake()` in order to avoid computing all partitions in case only a small number of rows were requested. This patch preserves this optimization by treating logical `Limit` operators specially when they appear as the terminal operator in a query plan: if a `Limit` is the final operator, then we will plan a special `CollectLimit` physical operator which implements the old `take()`-based logic.\n\nIn order to be able to match on operators only at the root of the query plan, this patch introduces a special `ReturnAnswer` logical operator which functions similar to `BroadcastHint`: this dummy operator is inserted at the root of the optimized logical plan before invoking the physical planner, allowing the planner to pattern-match on it.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #7334 from JoshRosen/remove-copy-in-limit.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/06f0df6df204c4722ff8a6bf909abaa32a715c41"},{"sha":"8e4d15f70713e1aaaa96dfb3ea4ccc5bb08eb2ce","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13101][SQL] nullability of array type element should not fail analysis of encoder\n\nnullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #11035 from cloud-fan/ignore-nullability.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/8e4d15f70713e1aaaa96dfb3ea4ccc5bb08eb2ce"},{"sha":"37bc203c8dd5022cb11d53b697c28a737ee85bcc","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13210][SQL] catch OOM when allocate memory and expand array\n\nThere is a bug when we try to grow the buffer, OOM is ignore wrongly (the assert also skipped by JVM), then we try grow the array again, this one will trigger spilling free the current page, the current record we inserted will be invalid.\n\nThe root cause is that JVM has less free memory than MemoryManager thought, it will OOM when allocate a page without trigger spilling. We should catch the OOM, and acquire memory again to trigger spilling.\n\nAnd also, we could not grow the array in `insertRecord` of `InMemorySorter` (it was there just for easy testing).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11095 from davies/fix_expand.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/37bc203c8dd5022cb11d53b697c28a737ee85bcc"},{"sha":"ff0af0ddfa4d198b203c3a39f8532cfbd4f4e027","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13095] [SQL] improve performance for broadcast join with dimension table\n\nThis PR improve the performance for Broadcast join with dimension tables, which is common in data warehouse.\n\nIf the join key can fit in a long, we will use a special api `get(Long)` to get the rows from HashedRelation.\n\nIf the HashedRelation only have unique keys, we will use a special api `getValue(Long)` or `getValue(InternalRow)`.\n\nIf the keys can fit within a long, also the keys are dense, we will use a array of UnsafeRow, instead a hash map.\n\nTODO: will do cleanup\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11065 from davies/gen_dim.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/ff0af0ddfa4d198b203c3a39f8532cfbd4f4e027"},{"sha":"eeaf45b92695c577279f3a17d8c80ee40425e9aa","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-10620][SPARK-13054] Minor addendum to #10835\n\nAdditional changes to #10835, mainly related to style and visibility. This patch also adds back a few deprecated methods for backward compatibility.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10958 from andrewor14/task-metrics-to-accums-followups.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/eeaf45b92695c577279f3a17d8c80ee40425e9aa"},{"sha":"3708d13f1a282a9ebf12e3b736f1aa1712cbacd5","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-12992] [SQL] Support vectorized decoding in UnsafeRowParquetRecordReader.\n\nWIP: running tests. Code needs a bit of clean up.\n\nThis patch completes the vectorized decoding with the goal of passing the existing\ntests. There is still more patches to support the rest of the format spec, even\njust for flat schemas.\n\nThis patch adds a new flag to enable the vectorized decoding. Tests were updated\nto try with both modes where applicable.\n\nOnce this is working well, we can remove the previous code path.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #11055 from nongli/spark-12992-2.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/3708d13f1a282a9ebf12e3b736f1aa1712cbacd5"},{"sha":"f9307d8fc5223b4c5be07e3dc691a327f3bbfa7f","author":{"email":"6c68cabc7124e43c9b4d50f86510600f69e74c99@odersky.com","name":"Jakob Odersky"},"message":"[SPARK-13176][CORE] Use native file linking instead of external process ln\n\nSince Spark requires at least JRE 1.7, it is safe to use built-in java.nio.Files.\n\nAuthor: Jakob Odersky <jakob@odersky.com>\n\nCloses #11098 from jodersky/SPARK-13176.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/f9307d8fc5223b4c5be07e3dc691a327f3bbfa7f"},{"sha":"159198eff67ee9ead08fba60a585494ea1575147","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-13165][STREAMING] Replace deprecated synchronizedBuffer in streaming\n\nBuilding with Scala 2.11 results in the warning trait SynchronizedBuffer in package mutable is deprecated: Synchronization via traits is deprecated as it is inherently unreliable. Consider java.util.concurrent.ConcurrentLinkedQueue as an alternative - we already use ConcurrentLinkedQueue elsewhere so lets replace it.\n\nSome notes about how behaviour is different for reviewers:\nThe Seq from a SynchronizedBuffer that was implicitly converted would continue to receive updates - however when we do the same conversion explicitly on the ConcurrentLinkedQueue this isn't the case. Hence changing some of the (internal & test) APIs to pass an Iterable. toSeq is safe to use if there are no more updates.\n\nAuthor: Holden Karau <holden@us.ibm.com>\nAuthor: tedyu <yuzhihong@gmail.com>\n\nCloses #11067 from holdenk/SPARK-13165-replace-deprecated-synchronizedBuffer-in-streaming.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/159198eff67ee9ead08fba60a585494ea1575147"},{"sha":"ce83fe9756582e73ada21c3741d15aa9bbf385ed","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-13201][SPARK-13200] Deprecation warning cleanups: KMeans & MFDataGenerator\n\nKMeans:\nMake a private non-deprecated version of setRuns API so that we can call it from the PythonAPI without deprecation warnings in our own build. Also use it internally when being called from train. Add a logWarning for non-1 values\n\nMFDataGenerator:\nApparently we are calling round on an integer which now in Scala 2.11 results in a warning (it didn't make any sense before either). Figure out if this is a mistake we can just remove or if we got the types wrong somewhere.\n\nI put these two together since they are both deprecation fixes in MLlib and pretty small, but I can split them up if we would prefer it that way.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #11112 from holdenk/SPARK-13201-non-deprecated-setRuns-SPARK-mathround-integer.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/ce83fe9756582e73ada21c3741d15aa9bbf385ed"},{"sha":"c882ec57de509895706dcafea8234238e4277a2d","author":{"email":"668e2a3febf6a4b75056e840735926407962bc4e@gmail.com","name":"Sebastián Ramírez"},"message":"[SPARK-13040][DOCS] Update JDBC deprecated SPARK_CLASSPATH documentation\n\nUpdate JDBC documentation based on http://stackoverflow.com/a/30947090/219530 as SPARK_CLASSPATH is deprecated.\n\nAlso, that's how it worked, it didn't work with the SPARK_CLASSPATH or the --jars alone.\n\nThis would solve issue: https://issues.apache.org/jira/browse/SPARK-13040\n\nAuthor: Sebastián Ramírez <tiangolo@gmail.com>\n\nCloses #10948 from tiangolo/patch-docs-jdbc.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/c882ec57de509895706dcafea8234238e4277a2d"},{"sha":"d9ba4d27f4d324a7055b9b914c75d176f3e2f71d","author":{"email":"78d5535267fa42b0e9b8f00a9e4cb8d5aa22b74f@gmail.com","name":"sachin aggarwal"},"message":"[SPARK-13177][EXAMPLES] Update ActorWordCount example to not directly use low level linked list as it is deprecated.\n\nAuthor: sachin aggarwal <different.sachin@gmail.com>\n\nCloses #11113 from agsachin/master.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/d9ba4d27f4d324a7055b9b914c75d176f3e2f71d"},{"sha":"e30121afac35439be5d42c04da6f047f7d973dd6","author":{"email":"53fdba43521c8d2c2b4ff3a6f81e9f27ad86e7ac@gmail.com","name":"Iulian Dragos"},"message":"[SPARK-13086][SHELL] Use the Scala REPL settings, to enable things like `-i file`.\n\nNow:\n\n```\n$ bin/spark-shell -i test.scala\nNOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.\nSetting default log level to \"WARN\".\nTo adjust logging level use sc.setLogLevel(newLevel).\n16/01/29 17:37:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n16/01/29 17:37:39 INFO Main: Created spark context..\nSpark context available as sc (master = local[*], app id = local-1454085459000).\n16/01/29 17:37:39 INFO Main: Created sql context..\nSQL context available as sqlContext.\nLoading test.scala...\nhello\n\nWelcome to\n      ____              __\n     / __/__  ___ _____/ /__\n    _\\ \\/ _ \\/ _ `/ __/  '_/\n   /___/ .__/\\_,_/_/ /_/\\_\\   version 2.0.0-SNAPSHOT\n      /_/\n\nUsing Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45)\nType in expressions to have them evaluated.\nType :help for more information.\n```\n\nAuthor: Iulian Dragos <jaguarul@gmail.com>\n\nCloses #10984 from dragos/issue/repl-eval-file.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/e30121afac35439be5d42c04da6f047f7d973dd6"},{"sha":"68ed3632c56389ab3ff4ea5d73c575f224dab4f6","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-13170][STREAMING] Investigate replacing SynchronizedQueue as it is deprecated\n\nReplace SynchronizeQueue with synchronized access to a Queue\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #11111 from srowen/SPARK-13170.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/68ed3632c56389ab3ff4ea5d73c575f224dab4f6"},{"sha":"34d0b70b309f16af263eb4e6d7c36e2ea170bc67","author":{"email":"ec969b9b5c2f2a49a0d9175cec61f09554863a3d@hortonworks.com","name":"Steve Loughran"},"message":"[SPARK-12807][YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3\n\nPatch to\n\n1. Shade jackson 2.x in spark-yarn-shuffle JAR: core, databind, annotation\n2. Use maven antrun to verify the JAR has the renamed classes\n\nBeing Maven-based, I don't know if the verification phase kicks in on an SBT/jenkins build. It will on a `mvn install`\n\nAuthor: Steve Loughran <stevel@hortonworks.com>\n\nCloses #10780 from steveloughran/stevel/patches/SPARK-12807-master-shuffle.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/34d0b70b309f16af263eb4e6d7c36e2ea170bc67"},{"sha":"2dbb9164405d6f595905c7d4b32e20177f0f669f","author":{"email":"00d7a42e7227da37af994b379e3eb0eed5cbdaa7@apache.org","name":"Luciano Resende"},"message":"[SPARK-13189] Cleanup build references to Scala 2.10\n\nAuthor: Luciano Resende <lresende@apache.org>\n\nCloses #11092 from lresende/SPARK-13189.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/2dbb9164405d6f595905c7d4b32e20177f0f669f"},{"sha":"7fe4fe630a3fc9755ebd0325bb595d76381633e8","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12888] [SQL] [FOLLOW-UP] benchmark the new hash expression\n\nAdds the benchmark results as comments.\n\nThe codegen version is slower than the interpreted version for `simple` case becasue of 3 reasons:\n\n1. codegen version use a more complex hash algorithm than interpreted version, i.e. `Murmur3_x86_32.hashInt` vs [simple multiplication and addition](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala#L153).\n2. codegen version will write the hash value to a row first and then read it out. I tried to create a `GenerateHasher` that can generate code to return hash value directly and got about 60% speed up for the `simple` case, does it worth?\n3. the row in `simple` case only has one int field, so the runtime reflection may be removed because of branch prediction, which makes the interpreted version faster.\n\nThe `array` case is also slow for similar reasons, e.g. array elements are of same type, so interpreted version can probably get rid of runtime reflection by branch prediction.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10917 from cloud-fan/hash-benchmark.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/7fe4fe630a3fc9755ebd0325bb595d76381633e8"},{"sha":"fae830d15846f7ffdfe49eeb45e175a3cdd2c670","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-13245][CORE] Call shuffleMetrics methods only in one thread for ShuffleBlockFetcherIterator\n\nCall shuffleMetrics's incRemoteBytesRead and incRemoteBlocksFetched when polling FetchResult from `results` so as to always use shuffleMetrics in one thread.\n\nAlso fix a race condition that could cause memory leak.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11138 from zsxwing/SPARK-13245.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/fae830d15846f7ffdfe49eeb45e175a3cdd2c670"},{"sha":"0e5ebac3c1f1ff58f938be59c7c9e604977d269c","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12950] [SQL] Improve lookup of BytesToBytesMap in aggregate\n\nThis PR improve the lookup of BytesToBytesMap by:\n\n1. Generate code for calculate the hash code of grouping keys.\n\n2. Do not use MemoryLocation, fetch the baseObject and offset for key and value directly (remove the indirection).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11010 from davies/gen_map.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/0e5ebac3c1f1ff58f938be59c7c9e604977d269c"},{"sha":"9267bc68fab65c6a798e065a1dbe0f5171df3077","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-10524][ML] Use the soft prediction to order categories' bins\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-10524\n\nCurrently we use the hard prediction (`ImpurityCalculator.predict`) to order categories' bins. But we should use the soft prediction.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #8734 from viirya/dt-soft-centroids.","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/9267bc68fab65c6a798e065a1dbe0f5171df3077"}]}