{"push_id":1178296654,"size":26,"distinct_size":1,"ref":"refs/heads/SPARK-16140","head":"70c312f683159b63089c25cf62dfab1074305191","before":"426c6593d382e50235664360713120363cb01492","commits":[{"sha":"d48935400ca47275f677b527c636976af09332c8","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16077] [PYSPARK] catch the exception from pickle.whichmodule()\n\n## What changes were proposed in this pull request?\n\nIn the case that we don't know which module a object came from, will call pickle.whichmodule() to go throught all the loaded modules to find the object, which could fail because some modules, for example, six, see https://bitbucket.org/gutworth/six/issues/63/importing-six-breaks-pickling\n\nWe should ignore the exception here, use `__main__` as the module name (it means we can't find the module).\n\n## How was this patch tested?\n\nManual tested. Can't have a unit test for this.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13788 from davies/whichmodule.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/d48935400ca47275f677b527c636976af09332c8"},{"sha":"5f8de21606f63408058bda44701d49fbf5027820","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SQL][MINOR] Simplify data source predicate filter translation.\n\n## What changes were proposed in this pull request?\nThis is a small patch to rewrite the predicate filter translation in DataSourceStrategy. The original code used excessive functional constructs (e.g. unzip) and was very difficult to understand.\n\n## How was this patch tested?\nShould be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13889 from rxin/simplify-predicate-filter.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/5f8de21606f63408058bda44701d49fbf5027820"},{"sha":"4435de1bd36e2c30b764725fae05a08733f4aad4","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16179][PYSPARK] fix bugs for Python udf in generate\n\n## What changes were proposed in this pull request?\n\nThis PR fix the bug when Python UDF is used in explode (generator), GenerateExec requires that all the attributes in expressions should be resolvable from children when creating, we should replace the children first, then replace it's expressions.\n\n```\n>>> df.select(explode(f(*df))).show()\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"/home/vlad/dev/spark/python/pyspark/sql/dataframe.py\", line 286, in show\n    print(self._jdf.showString(n, truncate))\n  File \"/home/vlad/dev/spark/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py\", line 933, in __call__\n  File \"/home/vlad/dev/spark/python/pyspark/sql/utils.py\", line 63, in deco\n    return f(*a, **kw)\n  File \"/home/vlad/dev/spark/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py\", line 312, in get_return_value\npy4j.protocol.Py4JJavaError: An error occurred while calling o52.showString.\n: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, tree:\nGenerate explode(<lambda>(_1#0L)), false, false, [col#15L]\n+- Scan ExistingRDD[_1#0L]\n\n\tat org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:50)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:387)\n\tat org.apache.spark.sql.execution.SparkPlan.makeCopy(SparkPlan.scala:69)\n\tat org.apache.spark.sql.execution.SparkPlan.makeCopy(SparkPlan.scala:45)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDown(QueryPlan.scala:177)\n\tat org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:144)\n\tat org.apache.spark.sql.execution.python.ExtractPythonUDFs$.org$apache$spark$sql$execution$python$ExtractPythonUDFs$$extract(ExtractPythonUDFs.scala:153)\n\tat org.apache.spark.sql.execution.python.ExtractPythonUDFs$$anonfun$apply$2.applyOrElse(ExtractPythonUDFs.scala:114)\n\tat org.apache.spark.sql.execution.python.ExtractPythonUDFs$$anonfun$apply$2.applyOrElse(ExtractPythonUDFs.scala:113)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:301)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:301)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:300)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:298)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:298)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:321)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:319)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:298)\n\tat org.apache.spark.sql.execution.python.ExtractPythonUDFs$.apply(ExtractPythonUDFs.scala:113)\n\tat org.apache.spark.sql.execution.python.ExtractPythonUDFs$.apply(ExtractPythonUDFs.scala:93)\n\tat org.apache.spark.sql.execution.QueryExecution$$anonfun$prepareForExecution$1.apply(QueryExecution.scala:95)\n\tat org.apache.spark.sql.execution.QueryExecution$$anonfun$prepareForExecution$1.apply(QueryExecution.scala:95)\n\tat scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)\n\tat scala.collection.immutable.List.foldLeft(List.scala:84)\n\tat org.apache.spark.sql.execution.QueryExecution.prepareForExecution(QueryExecution.scala:95)\n\tat org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:85)\n\tat org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:85)\n\tat org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2557)\n\tat org.apache.spark.sql.Dataset.head(Dataset.scala:1923)\n\tat org.apache.spark.sql.Dataset.take(Dataset.scala:2138)\n\tat org.apache.spark.sql.Dataset.showString(Dataset.scala:239)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:280)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:211)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.reflect.InvocationTargetException\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$13.apply(TreeNode.scala:413)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$13.apply(TreeNode.scala:413)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:412)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:387)\n\tat org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)\n\t... 42 more\nCaused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree: pythonUDF0#20\n\tat org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:50)\n\tat org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:88)\n\tat org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:87)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:279)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:279)\n\tat org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:278)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:321)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:319)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:284)\n\tat org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:268)\n\tat org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:87)\n\tat org.apache.spark.sql.execution.GenerateExec.<init>(GenerateExec.scala:63)\n\t... 52 more\nCaused by: java.lang.RuntimeException: Couldn't find pythonUDF0#20 in [_1#0L]\n\tat scala.sys.package$.error(package.scala:27)\n\tat org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:94)\n\tat org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:88)\n\tat org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)\n\t... 67 more\n```\n\n## How was this patch tested?\n\nAdded regression tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13883 from davies/udf_in_generate.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/4435de1bd36e2c30b764725fae05a08733f4aad4"},{"sha":"a65bcbc27dcd9b3053cb13c5d67251c8d48f4397","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16186] [SQL] Support partition batch pruning with `IN` predicate in InMemoryTableScanExec\n\n## What changes were proposed in this pull request?\n\nOne of the most frequent usage patterns for Spark SQL is using **cached tables**. This PR improves `InMemoryTableScanExec` to handle `IN` predicate efficiently by pruning partition batches. Of course, the performance improvement varies over the queries and the datasets. But, for the following simple query, the query duration in Spark UI goes from 9 seconds to 50~90ms. It's about over 100 times faster.\n\n**Before**\n```scala\n$ bin/spark-shell --driver-memory 6G\nscala> val df = spark.range(2000000000)\nscala> df.createOrReplaceTempView(\"t\")\nscala> spark.catalog.cacheTable(\"t\")\nscala> sql(\"select id from t where id = 1\").collect()    // About 2 mins\nscala> sql(\"select id from t where id = 1\").collect()    // less than 90ms\nscala> sql(\"select id from t where id in (1,2,3)\").collect()  // 9 seconds\n```\n\n**After**\n```scala\nscala> sql(\"select id from t where id in (1,2,3)\").collect() // less than 90ms\n```\n\nThis PR has impacts over 35 queries of TPC-DS if the tables are cached.\nNote that this optimization is applied for `IN`.  To apply `IN` predicate having more than 10 items, `spark.sql.optimizer.inSetConversionThreshold` option should be increased.\n\n## How was this patch tested?\n\nPass the Jenkins tests (including new testcases).\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13887 from dongjoon-hyun/SPARK-16186.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/a65bcbc27dcd9b3053cb13c5d67251c8d48f4397"},{"sha":"20768dade2fee5dfe967a4629cad477e3d3bce6e","author":{"name":"Davies Liu","email":"0974b7f8fcee704ff653a2542bba493bdfd47646@gmail.com"},"message":"Revert \"[SPARK-16186] [SQL] Support partition batch pruning with `IN` predicate in InMemoryTableScanExec\"\n\nThis reverts commit a65bcbc27dcd9b3053cb13c5d67251c8d48f4397.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/20768dade2fee5dfe967a4629cad477e3d3bce6e"},{"sha":"e5d0928e2473d1838ff5420c6a8964557c33135e","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16173] [SQL] Can't join describe() of DataFrame in Scala 2.10\n\n## What changes were proposed in this pull request?\n\nThis PR fixes `DataFrame.describe()` by forcing materialization to make the `Seq` serializable. Currently, `describe()` of DataFrame throws `Task not serializable` Spark exceptions when joining in Scala 2.10.\n\n## How was this patch tested?\n\nManual. (After building with Scala 2.10, test on `bin/spark-shell` and `bin/pyspark`.)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13900 from dongjoon-hyun/SPARK-16173.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/e5d0928e2473d1838ff5420c6a8964557c33135e"},{"sha":"9053054c7f5ec2b9e3d8efbe6bfbfa68a6d1f0d0","author":{"name":"Dilip Biswal","email":"6ab79ed79ae7ed4bf1dfe07d1c3e4c433e1f76a0@us.ibm.com"},"message":"[SPARK-16195][SQL] Allow users to specify empty over clause in window expressions through dataset API\n\n## What changes were proposed in this pull request?\nAllow to specify empty over clause in window expressions through dataset API\n\nIn SQL, its allowed to specify an empty OVER clause in the window expression.\n\n```SQL\nselect area, sum(product) over () as c from windowData\nwhere product > 3 group by area, product\nhaving avg(month) > 0 order by avg(month), product\n```\nIn this case the analytic function sum is presented based on all the rows of the result set\n\nCurrently its not allowed through dataset API and is handled in this PR.\n\n## How was this patch tested?\n\nAdded a new test in DataframeWindowSuite\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #13897 from dilipbiswal/spark-empty-over.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/9053054c7f5ec2b9e3d8efbe6bfbfa68a6d1f0d0"},{"sha":"d2e44d7db82ff3c3326af7bf7ea69c803803698e","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SPARK-16192][SQL] Add type checks in CollectSet\n\n## What changes were proposed in this pull request?\n`CollectSet` cannot have map-typed data because MapTypeData does not implement `equals`.\nSo, this pr is to add type checks in `CheckAnalysis`.\n\n## How was this patch tested?\nAdded tests to check failures when we found map-typed data in `CollectSet`.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #13892 from maropu/SPARK-16192.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/d2e44d7db82ff3c3326af7bf7ea69c803803698e"},{"sha":"a7d29499dca5b86e776abc225ece84391f09353a","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16186] [SQL] Support partition batch pruning with `IN` predicate in InMemoryTableScanExec\n\n## What changes were proposed in this pull request?\n\nOne of the most frequent usage patterns for Spark SQL is using **cached tables**. This PR improves `InMemoryTableScanExec` to handle `IN` predicate efficiently by pruning partition batches. Of course, the performance improvement varies over the queries and the datasets. But, for the following simple query, the query duration in Spark UI goes from 9 seconds to 50~90ms. It's about over 100 times faster.\n\n**Before**\n```scala\n$ bin/spark-shell --driver-memory 6G\nscala> val df = spark.range(2000000000)\nscala> df.createOrReplaceTempView(\"t\")\nscala> spark.catalog.cacheTable(\"t\")\nscala> sql(\"select id from t where id = 1\").collect()    // About 2 mins\nscala> sql(\"select id from t where id = 1\").collect()    // less than 90ms\nscala> sql(\"select id from t where id in (1,2,3)\").collect()  // 9 seconds\n```\n\n**After**\n```scala\nscala> sql(\"select id from t where id in (1,2,3)\").collect() // less than 90ms\n```\n\nThis PR has impacts over 35 queries of TPC-DS if the tables are cached.\nNote that this optimization is applied for `IN`.  To apply `IN` predicate having more than 10 items, `spark.sql.optimizer.inSetConversionThreshold` option should be increased.\n\n## How was this patch tested?\n\nPass the Jenkins tests (including new testcases).\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13887 from dongjoon-hyun/SPARK-16186.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/a7d29499dca5b86e776abc225ece84391f09353a"},{"sha":"a3c7b4187bad00dad87df7e3b5929a44d29568ed","author":{"name":"José Antonio","email":"06b4be0314d206f0ae6b9a0f424595e4d89551d2@gmail.com"},"message":"[MLLIB] org.apache.spark.mllib.util.SVMDataGenerator generates ArrayIndexOutOfBoundsException. I have found the bug and tested the solution.\n\n## What changes were proposed in this pull request?\n\nJust adjust the size of an array in line 58 so it does not cause an ArrayOutOfBoundsException in line 66.\n\n## How was this patch tested?\n\nManual tests. I have recompiled the entire project with the fix, it has been built successfully and I have run the code, also with good results.\n\nline 66: val yD = blas.ddot(trueWeights.length, x, 1, trueWeights, 1) + rnd.nextGaussian() * 0.1\ncrashes because trueWeights has length \"nfeatures + 1\" while \"x\" has length \"features\", and they should have the same length.\n\nTo fix this just make trueWeights be the same length as x.\n\nI have recompiled the project with the change and it is working now:\n[spark-1.6.1]$ spark-submit --master local[*] --class org.apache.spark.mllib.util.SVMDataGenerator mllib/target/spark-mllib_2.11-1.6.1.jar local /home/user/test\n\nAnd it generates the data successfully now in the specified folder.\n\nAuthor: José Antonio <joseanmunoz@gmail.com>\n\nCloses #13895 from j4munoz/patch-2.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/a3c7b4187bad00dad87df7e3b5929a44d29568ed"},{"sha":"bf665a958631125a1670504ef5966ef1a0e14798","author":{"name":"Sital Kedia","email":"e4343f82e25883a7c046475fe9553138fac5aa32@fb.com"},"message":"[SPARK-15958] Make initial buffer size for the Sorter configurable\n\n## What changes were proposed in this pull request?\n\nCurrently the initial buffer size in the sorter is hard coded inside the code and is too small for large workload. As a result, the sorter spends significant time expanding the buffer size and copying the data. It would be useful to have it configurable.\n\n## How was this patch tested?\n\nTested by running a job on the cluster.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #13699 from sitalkedia/config_sort_buffer_upstream.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/bf665a958631125a1670504ef5966ef1a0e14798"},{"sha":"3ee9695d1fcf3750cbf7896a56f8a1ba93f4e82f","author":{"name":"Alex Bozarth","email":"850363088eed533ef66882756a87f03f1fc98fde@us.ibm.com"},"message":"[SPARK-1301][WEB UI] Added anchor links to Accumulators and Tasks on StagePage\n\n## What changes were proposed in this pull request?\n\nSometimes the \"Aggregated Metrics by Executor\" table on the Stage page can get very long so actor links to the Accumulators and Tasks tables below it have been added to the summary at the top of the page. This has been done in the same way as the Jobs and Stages pages. Note: the Accumulators link only displays when the table exists.\n\n## How was this patch tested?\n\nManually Tested and dev/run-tests\n\n![justtasks](https://cloud.githubusercontent.com/assets/13952758/15165269/6e8efe8c-16c9-11e6-9784-cffe966fdcf0.png)\n![withaccumulators](https://cloud.githubusercontent.com/assets/13952758/15165270/7019ec9e-16c9-11e6-8649-db69ed7a317d.png)\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #13037 from ajbozarth/spark1301.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/3ee9695d1fcf3750cbf7896a56f8a1ba93f4e82f"},{"sha":"e87741589a24821b5fe73e5d9ee2164247998580","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16193][TESTS] Address flaky ExternalAppendOnlyMapSuite spilling tests\n\n## What changes were proposed in this pull request?\n\nMake spill tests wait until job has completed before returning the number of stages that spilled\n\n## How was this patch tested?\n\nExisting Jenkins tests.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #13896 from srowen/SPARK-16193.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/e87741589a24821b5fe73e5d9ee2164247998580"},{"sha":"30b182bcc088aef161585211c517f473b9ee6632","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-16184][SPARKR] conf API for SparkSession\n\n## What changes were proposed in this pull request?\n\nAdd `conf` method to get Runtime Config from SparkSession\n\n## How was this patch tested?\n\nunit tests, manual tests\n\nThis is how it works in sparkR shell:\n```\n SparkSession available as 'spark'.\n> conf()\n$hive.metastore.warehouse.dir\n[1] \"file:/opt/spark-2.0.0-bin-hadoop2.6/R/spark-warehouse\"\n\n$spark.app.id\n[1] \"local-1466749575523\"\n\n$spark.app.name\n[1] \"SparkR\"\n\n$spark.driver.host\n[1] \"10.0.2.1\"\n\n$spark.driver.port\n[1] \"45629\"\n\n$spark.executorEnv.LD_LIBRARY_PATH\n[1] \"$LD_LIBRARY_PATH:/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/jre/lib/amd64/server\"\n\n$spark.executor.id\n[1] \"driver\"\n\n$spark.home\n[1] \"/opt/spark-2.0.0-bin-hadoop2.6\"\n\n$spark.master\n[1] \"local[*]\"\n\n$spark.sql.catalogImplementation\n[1] \"hive\"\n\n$spark.submit.deployMode\n[1] \"client\"\n\n> conf(\"spark.master\")\n$spark.master\n[1] \"local[*]\"\n\n```\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13885 from felixcheung/rconf.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/30b182bcc088aef161585211c517f473b9ee6632"},{"sha":"b452026324da20f76f7d8b78e5ba1c007712e585","author":{"name":"杨浩","email":"84776c159e0fadc6718b867674befaa59dabe1fe@163.com"},"message":"[SPARK-16214][EXAMPLES] fix the denominator of SparkPi\n\n## What changes were proposed in this pull request?\n\nreduce the denominator of SparkPi by 1\n\n## How was this patch tested?\n\n  integration tests\n\nAuthor: 杨浩 <yanghaogn@163.com>\n\nCloses #13910 from yanghaogn/patch-1.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/b452026324da20f76f7d8b78e5ba1c007712e585"},{"sha":"52d4fe057909e8d431ae36f538dc4cafb351cdb5","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[MINOR][CORE] Fix display wrong free memory size in the log\n\n## What changes were proposed in this pull request?\n\nFree memory size displayed in the log is wrong (used memory), fix to make it correct.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #13804 from jerryshao/memory-log-fix.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/52d4fe057909e8d431ae36f538dc4cafb351cdb5"},{"sha":"11f420b4bbcd607346204fb6fd7db7efe948cdac","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-10591][SQL][TEST] Add a testcase to ensure if `checkAnswer` handles map correctly\n\n## What changes were proposed in this pull request?\n\nThis PR adds a testcase to ensure if `checkAnswer` handles Map type correctly.\n\n## How was this patch tested?\n\nPass the jenkins tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13913 from dongjoon-hyun/SPARK-10591.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/11f420b4bbcd607346204fb6fd7db7efe948cdac"},{"sha":"3e4e868c850e6b6da2c0d005167316e1abdc7460","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SPARK-16135][SQL] Remove hashCode and euqals in ArrayBasedMapData\n\n## What changes were proposed in this pull request?\nThis pr is to remove `hashCode` and `equals` in `ArrayBasedMapData` because the type cannot be used as join keys, grouping keys, or in equality tests.\n\n## How was this patch tested?\nAdd a new test suite `MapDataSuite` for comparison tests.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #13847 from maropu/UnsafeMapTest.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/3e4e868c850e6b6da2c0d005167316e1abdc7460"},{"sha":"c48c8ebc0aad433aab7af9e2ddf544d253ab9fd7","author":{"name":"Bill Chambers","email":"7d8fb5221dddb30ca0738503eceb5dce85cfc989@databricks.com"},"message":"[SPARK-16220][SQL] Revert Change to Bring Back SHOW FUNCTIONS Functionality\n\n## What changes were proposed in this pull request?\n\n- Fix tests regarding show functions functionality\n- Revert `catalog.ListFunctions` and `SHOW FUNCTIONS` to return to `Spark 1.X` functionality.\n\nCherry picked changes from this PR: https://github.com/apache/spark/pull/13413/files\n\n## How was this patch tested?\n\nUnit tests.\n\nAuthor: Bill Chambers <bill@databricks.com>\nAuthor: Bill Chambers <wchambers@ischool.berkeley.edu>\n\nCloses #13916 from anabranch/master.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/c48c8ebc0aad433aab7af9e2ddf544d253ab9fd7"},{"sha":"c17b1abff8f8c6d24cb0cf4ff4f8c14a780c64b0","author":{"name":"Yuhao Yang","email":"857769fa089c7426f7be1ac29063eaedcb246e23@intel.com"},"message":"[SPARK-16187][ML] Implement util method for ML Matrix conversion in scala/java\n\n## What changes were proposed in this pull request?\njira: https://issues.apache.org/jira/browse/SPARK-16187\nThis is to provide conversion utils between old/new vector columns in a DataFrame. So users can use it to migrate their datasets and pipelines manually.\n\n## How was this patch tested?\n\njava and scala ut\n\nAuthor: Yuhao Yang <yuhao.yang@intel.com>\n\nCloses #13888 from hhbyyh/matComp.","distinct":false,"url":"https://api.github.com/repos/keypointt/spark/commits/c17b1abff8f8c6d24cb0cf4ff4f8c14a780c64b0"}]}
{"actor":{"display_login":"keypointt"}}