{"push_id":1251739788,"size":10,"distinct_size":10,"ref":"refs/heads/master","head":"8fdc6ce400f9130399fbdd004df48b3ba95bcd6a","before":"2a3d286f3421f6836b71afcbda3084222752e6b1","commits":[{"sha":"1a028bdefa6312bf0eec46b89a1947da7e9d84af","author":{"name":"Stavros Kontopoulos","email":"42bafac589feb2200b51ee576d084d6a9af943fa@lightbend.com"},"message":"[SPARK-11714][MESOS] Make Spark on Mesos honor port restrictions on coarse grain mode\n\n- Make mesos coarse grained scheduler accept port offers and pre-assign ports\n\nPrevious attempt was for fine grained: https://github.com/apache/spark/pull/10808\n\nAuthor: Stavros Kontopoulos <stavros.kontopoulos@lightbend.com>\nAuthor: Stavros Kontopoulos <stavros.kontopoulos@typesafe.com>\n\nCloses #11157 from skonto/honour_ports_coarse.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/1a028bdefa6312bf0eec46b89a1947da7e9d84af"},{"sha":"ddf0d1e3fe18bcd01e1447feea1b76ce86087b3b","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[TRIVIAL][ML] Fix LogisticRegression typo in error message.\n\n## What changes were proposed in this pull request?\nFix ```LogisticRegression``` typo in error message.\n\n## How was this patch tested?\nDocs change, no new tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #14633 from yanboliang/lr-typo.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/ddf0d1e3fe18bcd01e1447feea1b76ce86087b3b"},{"sha":"3d8bfe7a39015c84cf95561fe17eb2808ce44084","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-16934][ML][MLLIB] Update LogisticCostAggregator serialization code to make it consistent with LinearRegression\n\n## What changes were proposed in this pull request?\n\nUpdate LogisticCostAggregator serialization code to make it consistent with #14109\n\n## How was this patch tested?\nMLlib 2.0:\n![image](https://cloud.githubusercontent.com/assets/19235986/17649601/5e2a79ac-61ee-11e6-833c-3bd8b5250470.png)\n\nAfter this PR:\n![image](https://cloud.githubusercontent.com/assets/19235986/17649599/52b002ae-61ee-11e6-9402-9feb3439880f.png)\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14520 from WeichenXu123/improve_logistic_regression_costfun.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/3d8bfe7a39015c84cf95561fe17eb2808ce44084"},{"sha":"564fe614c11deb657e0ac9e6b75e65370c48b7fe","author":{"name":"Junyang Qian","email":"d5766da94a207df74ee6bb6bd7d2e5654c64d4d7@databricks.com"},"message":"[SPARK-16508][SPARKR] Split docs for arrange and orderBy methods\n\n## What changes were proposed in this pull request?\n\nThis PR splits arrange and orderBy methods according to their functionality (the former for sorting sparkDataFrame and the latter for windowSpec).\n\n## How was this patch tested?\n\n![screen shot 2016-08-06 at 6 39 19 pm](https://cloud.githubusercontent.com/assets/15318264/17459969/51eade28-5c05-11e6-8ca1-8d8a8e344bab.png)\n![screen shot 2016-08-06 at 6 39 29 pm](https://cloud.githubusercontent.com/assets/15318264/17459966/51e3c246-5c05-11e6-8d35-3e905ca48676.png)\n![screen shot 2016-08-06 at 6 40 02 pm](https://cloud.githubusercontent.com/assets/15318264/17459967/51e650ec-5c05-11e6-8698-0f037f5199ff.png)\n\nAuthor: Junyang Qian <junyangq@databricks.com>\n\nCloses #14522 from junyangq/SPARK-16508-0.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/564fe614c11deb657e0ac9e6b75e65370c48b7fe"},{"sha":"5da6c4b24f512b63cd4e6ba7dd8968066a9396f5","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-16671][CORE][SQL] Consolidate code to do variable substitution.\n\nBoth core and sql have slightly different code that does variable substitution\nof config values. This change refactors that code and encapsulates the logic\nof reading config values and expading variables in a new helper class, which\ncan be configured so that both core and sql can use it without losing existing\nfunctionality, and allows for easier testing and makes it easier to add more\nfeatures in the future.\n\nTested with existing and new unit tests, and by running spark-shell with\nsome configs referencing variables and making sure it behaved as expected.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #14468 from vanzin/SPARK-16671.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/5da6c4b24f512b63cd4e6ba7dd8968066a9396f5"},{"sha":"fffb0c0d19a2444e7554dfe6b27de0c086112b17","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16700][PYSPARK][SQL] create DataFrame from dict/Row with schema\n\n## What changes were proposed in this pull request?\n\nIn 2.0, we verify the data type against schema for every row for safety, but with performance cost, this PR make it optional.\n\nWhen we verify the data type for StructType, it does not support all the types we support in infer schema (for example, dict), this PR fix that to make them consistent.\n\nFor Row object which is created using named arguments, the order of fields are sorted by name, they may be not different than the order in provided schema, this PR fix that by ignore the order of fields in this case.\n\n## How was this patch tested?\n\nCreated regression tests for them.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #14469 from davies/py_dict.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/fffb0c0d19a2444e7554dfe6b27de0c086112b17"},{"sha":"268b71d0d792f875fcfaec5314862236754a00d6","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-17065][SQL] Improve the error message when encountering an incompatible DataSourceRegister\n\n## What changes were proposed in this pull request?\n\nAdd an instruction to ask the user to remove or upgrade the incompatible DataSourceRegister in the error message.\n\n## How was this patch tested?\n\nTest command:\n```\nbuild/sbt -Dscala-2.10 package\nSPARK_SCALA_VERSION=2.10 bin/spark-shell --packages ai.h2o:sparkling-water-core_2.10:1.6.5\n\nscala> Seq(1).toDS().write.format(\"parquet\").save(\"foo\")\n```\n\nBefore:\n```\njava.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.h2o.DefaultSource could not be instantiated\n\tat java.util.ServiceLoader.fail(ServiceLoader.java:232)\n\tat java.util.ServiceLoader.access$100(ServiceLoader.java:185)\n\tat java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)\n\tat java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)\n\tat java.util.ServiceLoader$1.next(ServiceLoader.java:480)\n...\nCaused by: java.lang.NoClassDefFoundError: org/apache/spark/Logging\n\tat java.lang.ClassLoader.defineClass1(Native Method)\n\tat java.lang.ClassLoader.defineClass(ClassLoader.java:760)\n\tat java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\n\tat java.net.URLClassLoader.defineClass(URLClassLoader.java:467)\n\tat java.net.URLClassLoader.access$100(URLClassLoader.java:73)\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:368)\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:362)\n\tat java.security.AccessController.doPrivileged(Native Method)\n...\n```\n\nAfter:\n\n```\njava.lang.ClassNotFoundException: Detected an incompatible DataSourceRegister. Please remove the incompatible library from classpath or upgrade it. Error: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.h2o.DefaultSource could not be instantiated\n\tat org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:178)\n\tat org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:79)\n\tat org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:79)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:441)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:213)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:196)\n...\n```\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #14651 from zsxwing/SPARK-17065.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/268b71d0d792f875fcfaec5314862236754a00d6"},{"sha":"7de30d6e9e5d3020d2ba8c2ce08893d9cd822b56","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16916][SQL] serde/storage properties should not have limitations\n\n## What changes were proposed in this pull request?\n\n`CatalogStorageFormat.properties` can be used in 2 ways:\n\n1. for hive tables, it stores the serde properties.\n2. for data source tables, it stores the data source options, e.g. `path`, `skipHiveMetadata`, etc.\n\nhowever, both of them have nothing to do with data source properties, e.g. `spark.sql.sources.provider`, so they should not have limitations about data source properties.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14506 from cloud-fan/table-prop.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/7de30d6e9e5d3020d2ba8c2ce08893d9cd822b56"},{"sha":"7b65030e7a0af3a0bd09370fb069d659b36ff7f0","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-17034][SQL] adds expression UnresolvedOrdinal to represent the ordinals in GROUP BY or ORDER BY\n\n## What changes were proposed in this pull request?\n\nThis PR adds expression `UnresolvedOrdinal` to represent the ordinal in GROUP BY or ORDER BY, and fixes the rules when resolving ordinals.\n\nOrdinals in GROUP BY or ORDER BY like `1` in `order by 1` or `group by 1` should be considered as unresolved before analysis. But in current code, it uses `Literal` expression to store the ordinal. This is inappropriate as `Literal` itself is a resolved expression, it gives the user a wrong message that the ordinals has already been resolved.\n\n### Before this change\n\nOrdinal is stored as `Literal` expression\n\n```\nscala> sc.setLogLevel(\"TRACE\")\nscala> sql(\"select a from t group by 1 order by 1\")\n...\n'Sort [1 ASC], true\n +- 'Aggregate [1], ['a]\n     +- 'UnresolvedRelation `t\n```\n\nFor query:\n\n```\nscala> Seq(1).toDF(\"a\").createOrReplaceTempView(\"t\")\nscala> sql(\"select count(a), a from t group by 2 having a > 0\").show\n```\n\nDuring analysis, the intermediate plan before applying rule `ResolveAggregateFunctions` is:\n\n```\n'Filter ('a > 0)\n   +- Aggregate [2], [count(1) AS count(1)#83L, a#81]\n        +- LocalRelation [value#7 AS a#9]\n```\n\nBefore this PR, rule `ResolveAggregateFunctions` believes all expressions of `Aggregate` have already been resolved, and tries to resolve the expressions in `Filter` directly. But this is wrong, as ordinal `2` in Aggregate is not really resolved!\n\n### After this change\n\nOrdinals are stored as `UnresolvedOrdinal`.\n\n```\nscala> sc.setLogLevel(\"TRACE\")\nscala> sql(\"select a from t group by 1 order by 1\")\n...\n'Sort [unresolvedordinal(1) ASC], true\n +- 'Aggregate [unresolvedordinal(1)], ['a]\n      +- 'UnresolvedRelation `t`\n```\n\n## How was this patch tested?\n\nUnit tests.\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #14616 from clockfly/spark-16955.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/7b65030e7a0af3a0bd09370fb069d659b36ff7f0"},{"sha":"8fdc6ce400f9130399fbdd004df48b3ba95bcd6a","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-16964][SQL] Remove private[hive] from sql.hive.execution package\n\n## What changes were proposed in this pull request?\nThis PR is a small follow-up to https://github.com/apache/spark/pull/14554. This also widens the visibility of a few (similar) Hive classes.\n\n## How was this patch tested?\nNo test. Only a visibility change.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #14654 from hvanhovell/SPARK-16964-hive.","distinct":true,"url":"https://api.github.com/repos/srowen/spark/commits/8fdc6ce400f9130399fbdd004df48b3ba95bcd6a"}]}
{"actor":{"display_login":"srowen"}}