{"push_id":923381482,"size":60,"distinct_size":60,"ref":"refs/heads/master","head":"1cdc42d2b99edfec01066699a7620cca02b61f0e","before":"932cf44248e067ee7cae6fef79ddf2ab9b1c36d8","commits":[{"sha":"aa48164a43bd9ed9eab53fcacbed92819e84eaf7","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12495][SQL] use true as default value for propagateNull in NewInstance\n\nMost of cases we should propagate null when call `NewInstance`, and so far there is only one case we should stop null propagation: create product/java bean. So I think it makes more sense to propagate null by dafault.\n\nThis also fixes a bug when encode null array/map, which is firstly discovered in https://github.com/apache/spark/pull/10401\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10443 from cloud-fan/encoder.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/aa48164a43bd9ed9eab53fcacbed92819e84eaf7"},{"sha":"d1ca634db4ca9db7f0ba7ca38a0e03bcbfec23c9","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-12300] [SQL] [PYSPARK] fix schema inferance on local collections\n\nCurrent schema inference for local python collections halts as soon as there are no NullTypes. This is different than when we specify a sampling ratio of 1.0 on a distributed collection. This could result in incomplete schema information.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #10275 from holdenk/SPARK-12300-fix-schmea-inferance-on-local-collections.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/d1ca634db4ca9db7f0ba7ca38a0e03bcbfec23c9"},{"sha":"27a42c7108ced48a7f558990de2e4fc7ed340119","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests\n\nThis patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath.\n\nThis supplants the checks added in SPARK-4123 / #5093, which are currently disabled due to bugs.\n\nThis patch is based on pwendell's work in #8531.\n\nCloses #8531.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\nAuthor: Patrick Wendell <patrick@databricks.com>\n\nCloses #10461 from JoshRosen/SPARK-10359.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/27a42c7108ced48a7f558990de2e4fc7ed340119"},{"sha":"5c2682b0c8fd2aeae2af1adb716ee0d5f8b85135","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-12409][SPARK-12387][SPARK-12391][SQL] Support AND/OR/IN/LIKE push-down filters for JDBC\n\nThis is rework from #10386 and add more tests and LIKE push-down support.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #10468 from maropu/SupportMorePushdownInJdbc.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/5c2682b0c8fd2aeae2af1adb716ee0d5f8b85135"},{"sha":"b244297966be1d09f8e861cfe2d8e69f7bed84da","author":{"email":"dbe0629cf0118f7dffec38856cad0ec7ac880251@intel.com","name":"Carson Wang"},"message":"[SPARK-12399] Display correct error message when accessing REST API with an unknown app Id\n\nI got an exception when accessing the below REST API with an unknown application Id.\n`http://<server-url>:18080/api/v1/applications/xxx/jobs`\nInstead of an exception, I expect an error message \"no such app: xxx\" which is a similar error message when I access `/api/v1/applications/xxx`\n```\norg.spark-project.guava.util.concurrent.UncheckedExecutionException: java.util.NoSuchElementException: no app with key xxx\n\tat org.spark-project.guava.cache.LocalCache$Segment.get(LocalCache.java:2263)\n\tat org.spark-project.guava.cache.LocalCache.get(LocalCache.java:4000)\n\tat org.spark-project.guava.cache.LocalCache.getOrLoad(LocalCache.java:4004)\n\tat org.spark-project.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)\n\tat org.apache.spark.deploy.history.HistoryServer.getSparkUI(HistoryServer.scala:116)\n\tat org.apache.spark.status.api.v1.UIRoot$class.withSparkUI(ApiRootResource.scala:226)\n\tat org.apache.spark.deploy.history.HistoryServer.withSparkUI(HistoryServer.scala:46)\n\tat org.apache.spark.status.api.v1.ApiRootResource.getJobs(ApiRootResource.scala:66)\n```\n\nAuthor: Carson Wang <carson.wang@intel.com>\n\nCloses #10352 from carsonwang/unknownAppFix.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/b244297966be1d09f8e861cfe2d8e69f7bed84da"},{"sha":"f76ee109d87e727710d2721e4be47fdabc21582c","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-8641][SPARK-12455][SQL] Native Spark Window functions - Follow-up (docs & tests)\n\nThis PR is a follow-up for PR https://github.com/apache/spark/pull/9819. It adds documentation for the window functions and a couple of NULL tests.\n\nThe documentation was largely based on the documentation in (the source of)  Hive and Presto:\n* https://prestodb.io/docs/current/functions/window.html\n* https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics\n\nI am not sure if we need to add the licenses of these two projects to the licenses directory. They are both under the ASL. srowen any thoughts?\n\ncc yhuai\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #10402 from hvanhovell/SPARK-8641-docs.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/f76ee109d87e727710d2721e4be47fdabc21582c"},{"sha":"ee8f8d318417c514fbb26e57157483d466ddbfae","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12588] Remove HttpBroadcast in Spark 2.0.\n\nWe switched to TorrentBroadcast in Spark 1.1, and HttpBroadcast has been undocumented since then. It's time to remove it in Spark 2.0.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10531 from rxin/SPARK-12588.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/ee8f8d318417c514fbb26e57157483d466ddbfae"},{"sha":"fd3333313864e21e3a9c95577723c931357d1f16","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][YARN] Fix import ordering.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10536 from vanzin/SPARK-3873-yarn.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/fd3333313864e21e3a9c95577723c931357d1f16"},{"sha":"9140d9074379055a0b4b2f5c381362b31c141941","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][GRAPHX] Import order fixes.\n\nThere's one warning left, caused by a bug in the checker.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10537 from vanzin/SPARK-3873-graphx.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/9140d9074379055a0b4b2f5c381362b31c141941"},{"sha":"be33a0cd3def86e0aa64dab411e504abbbdfb03c","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12561] Remove JobLogger in Spark 2.0.\n\nIt was research code and has been deprecated since 1.0.0. No one really uses it since they can just use event logging.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10530 from rxin/SPARK-12561.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/be33a0cd3def86e0aa64dab411e504abbbdfb03c"},{"sha":"7b4452ba98d53ed646a2e744bb701702fc5371a7","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"House cleaning: close open pull requests created before June 1st, 2015\n\nCloses #5358\nCloses #3744\nCloses #3677\nCloses #3536\nCloses #3249\nCloses #3221\nCloses #2446\nCloses #3794\nCloses #3815\nCloses #3816\nCloses #3866\nCloses #4286\nCloses #5184\nCloses #5170\nCloses #5142\nCloses #5025\nCloses #5005\nCloses #4897\nCloses #4887\nCloses #4849\nCloses #4632\nCloses #4622\nCloses #4456\nCloses #4449\nCloses #4417\nCloses #5483\nCloses #5325\nCloses #6545\nCloses #6449\nCloses #6433\nCloses #6416\nCloses #6403\nCloses #6386\nCloses #6263\nCloses #6245\nCloses #6213\nCloses #6155\nCloses #6133\nCloses #6018\nCloses #5978\nCloses #5869\nCloses #5852\nCloses #5848\nCloses #5754\nCloses #5598\nCloses #5503\nCloses #4380","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/7b4452ba98d53ed646a2e744bb701702fc5371a7"},{"sha":"c642c3a2102fd016deabcb78bd0292bbf3dd7acf","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Closes #10386 since it was superseded by #10468.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/c642c3a2102fd016deabcb78bd0292bbf3dd7acf"},{"sha":"93b52abca708aba90b6d0d36c92b920f6f8338ad","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"House cleaning: close old pull requests.\n\nCloses #5400\nCloses #5408\nCloses #5423\nCloses #5668\nCloses #6757\nCloses #6745\nCloses #6613","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/93b52abca708aba90b6d0d36c92b920f6f8338ad"},{"sha":"e6c77874b915691dead91e8d96ad9f58ba3a73db","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12585] [SQL] move numFields to constructor of UnsafeRow\n\nRight now, numFields will be passed in by pointTo(), then bitSetWidthInBytes is calculated, making pointTo() a little bit heavy.\n\nIt should be part of constructor of UnsafeRow.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10528 from davies/numFields.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/e6c77874b915691dead91e8d96ad9f58ba3a73db"},{"sha":"4f5a24d7e73104771f233af041eeba4f41675974","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-7995][SPARK-6280][CORE] Remove AkkaRpcEnv and remove systemName from setupEndpointRef\n\n### Remove AkkaRpcEnv\n\nKeep `SparkEnv.actorSystem` because Streaming still uses it. Will remove it and AkkaUtils after refactoring Streaming actorStream API.\n\n### Remove systemName\nThere are 2 places using `systemName`:\n* `RpcEnvConfig.name`. Actually, although it's used as `systemName` in `AkkaRpcEnv`, `NettyRpcEnv` uses it as the service name to output the log `Successfully started service *** on port ***`. Since the service name in log is useful, I keep `RpcEnvConfig.name`.\n* `def setupEndpointRef(systemName: String, address: RpcAddress, endpointName: String)`. Each `ActorSystem` has a `systemName`. Akka requires `systemName` in its URI and will refuse a connection if `systemName` is not matched. However, `NettyRpcEnv` doesn't use it. So we can remove `systemName` from `setupEndpointRef` since we are removing `AkkaRpcEnv`.\n\n### Remove RpcEnv.uriOf\n\n`uriOf` exists because Akka uses different URI formats for with and without authentication, e.g., `akka.ssl.tcp...` and `akka.tcp://...`. But `NettyRpcEnv` uses the same format. So it's not necessary after removing `AkkaRpcEnv`.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10459 from zsxwing/remove-akka-rpc-env.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/4f5a24d7e73104771f233af041eeba4f41675974"},{"sha":"5cdecb1841f5f1208a6100a673a768c84396633f","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-12039][SQL] Re-enable HiveSparkSubmitSuite's SPARK-9757 Persist Parquet relation with decimal column\n\nhttps://issues.apache.org/jira/browse/SPARK-12039\n\nsince we do not support hadoop1, we can re-enable this test in master.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #10533 from yhuai/SPARK-12039-enable.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/5cdecb1841f5f1208a6100a673a768c84396633f"},{"sha":"efb10cc9ad370955cec64e8f63a3b646058a9840","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][STREAMING] Import order fixes for streaming.\n\nAlso included a few miscelaneous other modules that had very few violations.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10532 from vanzin/SPARK-3873-streaming.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/efb10cc9ad370955cec64e8f63a3b646058a9840"},{"sha":"5adec63a922d6f60cd6cb87ebdab61a17131ac1a","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.sh script\n\nThis patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in #10461):\n\n- Use `build/mvn --force` instead of `mvn` in one additional place.\n- Explicitly set a zero exit code on success.\n- Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/).\n- Set `should_run_build_tests=True` for `build` module (this somehow got lost).\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10543 from JoshRosen/dep-script-fixes.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/5adec63a922d6f60cd6cb87ebdab61a17131ac1a"},{"sha":"c9dbfcc653b868fdb28106c1d1bcb6cb6caac6cc","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11743][SQL] Move the test for arrayOfUDT\n\nA following pr for #9712. Move the test for arrayOfUDT.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #10538 from viirya/move-udt-test.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/c9dbfcc653b868fdb28106c1d1bcb6cb6caac6cc"},{"sha":"a59a357cae82ca9b6926b55903ce4f12ae131735","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][MLLIB] Import order fixes.\n\nA slight adjustment to the checker configuration was needed; there is\na handful of warnings still left, but those are because of a bug in\nthe checker that I'll fix separately (before enabling errors for the\nchecker, of course).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10535 from vanzin/SPARK-3873-mllib.","distinct":true,"url":"https://api.github.com/repos/iyounus/spark/commits/a59a357cae82ca9b6926b55903ce4f12ae131735"}]}