{"push_id":1323601995,"size":89,"distinct_size":89,"ref":"refs/heads/weighted_nb","head":"af53267c4cb64501a75e8a91eb276e513c48e0bb","before":"8742be32eb8100a0de9c0b6a419749b095e52c0c","commits":[{"sha":"0336ff1baf769774bf0bead3aefa12e416bb4f7e","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-17051][SQL] we should use hadoopConf in InsertIntoHiveTable\n\n## What changes were proposed in this pull request?\n\nHive confs in hive-site.xml will be loaded in `hadoopConf`, so we should use `hadoopConf` in `InsertIntoHiveTable` instead of `SessionState.conf`\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14634 from cloud-fan/bug.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/0336ff1baf769774bf0bead3aefa12e416bb4f7e"},{"sha":"d26feaf773cf2e6031c4aa7d9b1e5c4dd99adb5b","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-15698][SQL][STREAMING] Add the ability to remove the old MetadataLog in FileStreamSource\n\n## What changes were proposed in this pull request?\n\nCurrent `metadataLog` in `FileStreamSource` will add a checkpoint file in each batch but do not have the ability to remove/compact, which will lead to large number of small files when running for a long time. So here propose to compact the old logs into one file. This method is quite similar to `FileStreamSinkLog` but simpler.\n\n## How was this patch tested?\n\nUnit test added.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #13513 from jerryshao/SPARK-15698.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/d26feaf773cf2e6031c4aa7d9b1e5c4dd99adb5b"},{"sha":"379056e62fd137884fdd81ab47fe4e0fdb2d98b6","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-17549][SQL] Revert \"[] Only collect table size stat in driver for cached relation.\"\n\nThis reverts commit 39e2bad6a866d27c3ca594d15e574a1da3ee84cc because of the problem mentioned at https://issues.apache.org/jira/browse/SPARK-17549?focusedCommentId=15505060&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15505060\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #15157 from yhuai/revert-SPARK-17549.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/379056e62fd137884fdd81ab47fe4e0fdb2d98b6"},{"sha":"76be3b21ec7641b6901f5e054bdac6fe6d41b89a","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-17611][YARN][TEST] Make shuffle service test really test auth.\n\nCurrently, the code is just swallowing exceptions, and not really checking\nwhether the auth information was being recorded properly. Fix both problems,\nand also avoid tests inadvertently affecting other tests by modifying the\nshared config variable (by making it not shared).\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #15161 from vanzin/SPARK-17611.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/76be3b21ec7641b6901f5e054bdac6fe6d41b89a"},{"sha":"0a6bb7cf883b2fc8da199d297e5b5f75e088d304","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-17513][SQL] Make StreamExecution garbage-collect its metadata\n\n## What changes were proposed in this pull request?\nThis PR modifies StreamExecution such that it discards metadata for batches that have already been fully processed. I used the purge method that was added as part of SPARK-17235.\n\nThis is a resubmission of 15126, which was based on work by frreiss in #15067, but fixed the test case along with some typos.\n\n## How was this patch tested?\nA new test case in StreamingQuerySuite. The test case would fail without the changes in this pull request.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #15166 from petermaxlee/SPARK-17513-2.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/0a6bb7cf883b2fc8da199d297e5b5f75e088d304"},{"sha":"24e2d4ec56457765bc93d988a70170ff12f2bcb2","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[MINOR][BUILD] Fix CheckStyle Error\n\n## What changes were proposed in this pull request?\nThis PR is to fix the code style errors before 2.0.1 release.\n\n## How was this patch tested?\nManual.\n\nBefore:\n```\n./dev/lint-java\nUsing `mvn` from path: /usr/local/bin/mvn\nCheckstyle checks failed at following occurrences:\n[ERROR] src/main/java/org/apache/spark/network/client/TransportClient.java:[153] (sizes) LineLength: Line is longer than 100 characters (found 107).\n[ERROR] src/main/java/org/apache/spark/network/client/TransportClient.java:[196] (sizes) LineLength: Line is longer than 100 characters (found 108).\n[ERROR] src/main/java/org/apache/spark/network/client/TransportClient.java:[239] (sizes) LineLength: Line is longer than 100 characters (found 115).\n[ERROR] src/main/java/org/apache/spark/network/server/TransportRequestHandler.java:[119] (sizes) LineLength: Line is longer than 100 characters (found 107).\n[ERROR] src/main/java/org/apache/spark/network/server/TransportRequestHandler.java:[129] (sizes) LineLength: Line is longer than 100 characters (found 104).\n[ERROR] src/main/java/org/apache/spark/network/util/LevelDBProvider.java:[124,11] (modifier) ModifierOrder: 'static' modifier out of order with the JLS suggestions.\n[ERROR] src/main/java/org/apache/spark/network/util/TransportConf.java:[26] (regexp) RegexpSingleline: No trailing whitespace allowed.\n[ERROR] src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java:[33] (sizes) LineLength: Line is longer than 100 characters (found 110).\n[ERROR] src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java:[38] (sizes) LineLength: Line is longer than 100 characters (found 110).\n[ERROR] src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java:[43] (sizes) LineLength: Line is longer than 100 characters (found 106).\n[ERROR] src/main/java/org/apache/spark/util/collection/unsafe/sort/PrefixComparators.java:[48] (sizes) LineLength: Line is longer than 100 characters (found 110).\n[ERROR] src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java:[0] (misc) NewlineAtEndOfFile: File does not end with a newline.\n[ERROR] src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java:[67] (sizes) LineLength: Line is longer than 100 characters (found 106).\n[ERROR] src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:[200] (regexp) RegexpSingleline: No trailing whitespace allowed.\n[ERROR] src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:[309] (regexp) RegexpSingleline: No trailing whitespace allowed.\n[ERROR] src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:[332] (regexp) RegexpSingleline: No trailing whitespace allowed.\n[ERROR] src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:[348] (regexp) RegexpSingleline: No trailing whitespace allowed.\n ```\nAfter:\n```\n./dev/lint-java\nUsing `mvn` from path: /usr/local/bin/mvn\nCheckstyle checks passed.\n```\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #15170 from Sherry302/fixjavastyle.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/24e2d4ec56457765bc93d988a70170ff12f2bcb2"},{"sha":"71e9e35512455a07ce07ad3f6abea84ac65aa1e9","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-15698][SQL][STREAMING][FOLLW-UP] Fix FileStream source and sink log get configuration issue\n\n## What changes were proposed in this pull request?\n\nThis issue was introduced in the previous commit of SPARK-15698. Mistakenly change the way to get configuration back to original one, so here with the follow up PR to revert them up.\n\n## How was this patch tested?\n\nN/A\n\nPing zsxwing , please review again, sorry to bring the inconvenience. Thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #15173 from jerryshao/SPARK-15698-follow.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/71e9e35512455a07ce07ad3f6abea84ac65aa1e9"},{"sha":"b40b77d87efd79ff84cd2cb922c96987f839b44f","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[CORE][DOC] Fix errors in comments\n\n## What changes were proposed in this pull request?\nWhile reading source code of CORE and SQL core, I found some minor errors in comments such as extra space, missing blank line and grammar error.\n\nI fixed these minor errors and might find more during my source code study.\n\n## How was this patch tested?\nManually build\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #15151 from wangmiao1981/mem.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/b40b77d87efd79ff84cd2cb922c96987f839b44f"},{"sha":"a052fc515ac877f06d4d1eee05ce6c2696ebfd24","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-17585][PYSPARK][CORE] PySpark SparkContext.addFile supports adding files recursively\n\n## What changes were proposed in this pull request?\nUsers would like to add a directory as dependency in some cases, they can use ```SparkContext.addFile``` with argument ```recursive=true``` to recursively add all files under the directory by using Scala. But Python users can only add file not directory, we should also make it supported.\n\n## How was this patch tested?\nUnit test.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #15140 from yanboliang/spark-17585.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/a052fc515ac877f06d4d1eee05ce6c2696ebfd24"},{"sha":"a209c3373549bf430848d4a0a369c32096c57465","author":{"name":"William Benton","email":"d5b78dab7a764904ac065aeb06649b6347ced923@redhat.com"},"message":"[SPARK-17595][MLLIB] Use a bounded priority queue to find synonyms in Word2VecModel\n\n## What changes were proposed in this pull request?\n\nThe code in `Word2VecModel.findSynonyms` to choose the vocabulary elements with the highest similarity to the query vector currently sorts the collection of similarities for every vocabulary element. This involves making multiple copies of the collection of similarities while doing a (relatively) expensive sort. It would be more efficient to find the best matches by maintaining a bounded priority queue and populating it with a single pass over the vocabulary, and that is exactly what this patch does.\n\n## How was this patch tested?\n\nThis patch adds no user-visible functionality and its correctness should be exercised by existing tests.  To ensure that this approach is actually faster, I made a microbenchmark for `findSynonyms`:\n\n```\nobject W2VTiming {\n  import org.apache.spark.{SparkContext, SparkConf}\n  import org.apache.spark.mllib.feature.Word2VecModel\n  def run(modelPath: String, scOpt: Option[SparkContext] = None) {\n    val sc = scOpt.getOrElse(new SparkContext(new SparkConf(true).setMaster(\"local[*]\").setAppName(\"test\")))\n    val model = Word2VecModel.load(sc, modelPath)\n    val keys = model.getVectors.keys\n    val start = System.currentTimeMillis\n    for(key <- keys) {\n      model.findSynonyms(key, 5)\n      model.findSynonyms(key, 10)\n      model.findSynonyms(key, 25)\n      model.findSynonyms(key, 50)\n    }\n    val finish = System.currentTimeMillis\n    println(\"run completed in \" + (finish - start) + \"ms\")\n  }\n}\n```\n\nI ran this test on a model generated from the complete works of Jane Austen and found that the new approach was over 3x faster than the old approach.  (If the `num` argument to `findSynonyms` is very close to the vocabulary size, the new approach will have less of an advantage over the old one.)\n\nAuthor: William Benton <willb@redhat.com>\n\nCloses #15150 from willb/SPARK-17595.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/a209c3373549bf430848d4a0a369c32096c57465"},{"sha":"71add7e5783f3f0292c26b4dd6a7d3292e958e70","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-17617][SQL] Remainder(%) expression.eval returns incorrect result on double value\n\n## What changes were proposed in this pull request?\n\nRemainder(%) expression's `eval()` returns incorrect result when the dividend is a big double. The reason is that Remainder converts the double dividend to decimal to do \"%\", and that lose precision.\n\nThis bug only affects the `eval()` that is used by constant folding, the codegen path is not impacted.\n\n### Before change\n```\nscala> -5083676433652386516D % 10\nres2: Double = -6.0\n\nscala> spark.sql(\"select -5083676433652386516D % 10 as a\").show\n+---+\n|  a|\n+---+\n|0.0|\n+---+\n```\n\n### After change\n```\nscala> spark.sql(\"select -5083676433652386516D % 10 as a\").show\n+----+\n|   a|\n+----+\n|-6.0|\n+----+\n```\n\n## How was this patch tested?\n\nUnit test.\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #15171 from clockfly/SPARK-17617.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/71add7e5783f3f0292c26b4dd6a7d3292e958e70"},{"sha":"307e6ea4603f536e314b74a0f0f15c043ea9c482","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-17599] Prevent ListingFileCatalog from failing if path doesn't exist\n\n## What changes were proposed in this pull request?\n\nThe `ListingFileCatalog` lists files given a set of resolved paths. If a folder is deleted at any time between the paths were resolved and the file catalog can check for the folder, the Spark job fails. This may abruptly stop long running StructuredStreaming jobs for example.\n\nFolders may be deleted by users or automatically by retention policies. These cases should not prevent jobs from successfully completing.\n\n## How was this patch tested?\n\nUnit test in `FileCatalogSuite`\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #15153 from brkyvz/SPARK-17599.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/307e6ea4603f536e314b74a0f0f15c043ea9c482"},{"sha":"bc7c239c52486f8d9d2de83681f937c7860b0190","author":{"name":"Peng, Meng","email":"6c4c4b336ba0af8858d16e0da367ee314642f197@intel.com"},"message":"[SPARK-17017][MLLIB][ML] add a chiSquare Selector based on False Positive Rate (FPR) test\n\n## What changes were proposed in this pull request?\n\nUnivariate feature selection works by selecting the best features based on univariate statistical tests. False Positive Rate (FPR) is a popular univariate statistical test for feature selection. We add a chiSquare Selector based on False Positive Rate (FPR) test in this PR, like it is implemented in scikit-learn.\nhttp://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection\n\n## How was this patch tested?\n\nAdd Scala ut\n\nAuthor: Peng, Meng <peng.meng@intel.com>\n\nCloses #14597 from mpjlu/fprChiSquare.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/bc7c239c52486f8d9d2de83681f937c7860b0190"},{"sha":"162eb0f72febb8fed3e1d2a42206690f36215b2c","author":{"name":"VinceShieh","email":"2445c9be3c24c11954fcdbf8bdec5f86733bb70d@intel.com"},"message":"[SPARK-17219][ML] Add NaN value handling in Bucketizer\n\n## What changes were proposed in this pull request?\nThis PR fixes an issue when Bucketizer is called to handle a dataset containing NaN value.\nSometimes, null value might also be useful to users, so in these cases, Bucketizer should\nreserve one extra bucket for NaN values, instead of throwing an illegal exception.\nBefore:\n```\nBucketizer.transform on NaN value threw an illegal exception.\n```\nAfter:\n```\nNaN values will be grouped in an extra bucket.\n```\n## How was this patch tested?\nNew test cases added in `BucketizerSuite`.\nSigned-off-by: VinceShieh <vincent.xieintel.com>\n\nAuthor: VinceShieh <vincent.xie@intel.com>\n\nCloses #14858 from VinceShieh/spark-17219.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/162eb0f72febb8fed3e1d2a42206690f36215b2c"},{"sha":"dbbcf88d106ec31316dd1c58f8253652bddea99c","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-17583][SQL] Remove uesless rowSeparator variable and set auto-expanding buffer as default for maxCharsPerColumn option in CSV\n\n## What changes were proposed in this pull request?\n\nThis PR includes the changes below:\n\n1. Upgrade Univocity library from 2.1.1 to 2.2.1\n\n  This includes some performance improvement and also enabling auto-extending buffer in `maxCharsPerColumn` option in CSV. Please refer the [release notes](https://github.com/uniVocity/univocity-parsers/releases).\n\n2. Remove useless `rowSeparator` variable existing in `CSVOptions`\n\n  We have this unused variable in [CSVOptions.scala#L127](https://github.com/apache/spark/blob/29952ed096fd2a0a19079933ff691671d6f00835/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala#L127) but it seems possibly causing confusion that it actually does not care of `\\r\\n`. For example, we have an issue open about this, [SPARK-17227](https://issues.apache.org/jira/browse/SPARK-17227), describing this variable.\n\n  This variable is virtually not being used because we rely on `LineRecordReader` in Hadoop which deals with only both `\\n` and `\\r\\n`.\n\n3. Set the default value of `maxCharsPerColumn` to auto-expending.\n\n  We are setting 1000000 for the length of each column. It'd be more sensible we allow auto-expending rather than fixed length by default.\n\n  To make sure, using `-1` is being described in the release note, [2.2.0](https://github.com/uniVocity/univocity-parsers/releases/tag/v2.2.0).\n\n## How was this patch tested?\n\nN/A\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15138 from HyukjinKwon/SPARK-17583.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/dbbcf88d106ec31316dd1c58f8253652bddea99c"},{"sha":"ab3684e66ea38e155954373cfe8db7d85fd741a8","author":{"name":"erenavsarogullari","email":"2b1f8430a64e745099cfadb513e73988eae74931@gmail.com"},"message":"[CORE][MINOR] Add minor code change to TaskState and Task\n\n## What changes were proposed in this pull request?\n- TaskState and ExecutorState expose isFailed and isFinished functions. It can be useful to add test coverage for different states. Currently, Other enums do not expose any functions so this PR aims just these two enums.\n- `private` access modifier is added for Finished Task States Set\n- A minor doc change is added.\n\n## How was this patch tested?\nNew Unit tests are added and run locally.\n\nAuthor: erenavsarogullari <erenavsarogullari@gmail.com>\n\nCloses #15143 from erenavsarogullari/SPARK-17584.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/ab3684e66ea38e155954373cfe8db7d85fd741a8"},{"sha":"be78d9903ab19106a869f001521ae7c5f79bd87c","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-17590][SQL] Analyze CTE definitions at once and allow CTE subquery to define CTE\n\n## What changes were proposed in this pull request?\n\nWe substitute logical plan with CTE definitions in the analyzer rule CTESubstitution. A CTE definition can be used in the logical plan for multiple times, and its analyzed logical plan should be the same. We should not analyze CTE definitions multiple times when they are reused in the query.\n\nBy analyzing CTE definitions before substitution, we can support defining CTE in subquery.\n\n## How was this patch tested?\n\nJenkins tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #15146 from viirya/cte-analysis-once.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/be78d9903ab19106a869f001521ae7c5f79bd87c"},{"sha":"2e9bed20c46adaff43abead654120e08cf61c0fe","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17418] Prevent kinesis-asl-assembly artifacts from being published\n\nThis patch updates the `kinesis-asl-assembly` build to prevent that module from being published as part of Maven releases and snapshot builds.\n\nThe `kinesis-asl-assembly` includes classes from the Kinesis Client Library (KCL) and Kinesis Producer Library (KPL), both of which are licensed under the Amazon Software License and are therefore prohibited from being distributed in Apache releases.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #15167 from JoshRosen/stop-publishing-kinesis-assembly.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/2e9bed20c46adaff43abead654120e08cf61c0fe"},{"sha":"4ece1bf5be47f11c22da1d32181d72a2a4184f79","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-11918][ML] Better error from WLS for cases like singular input\n\n## What changes were proposed in this pull request?\n\nUpdate error handling for Cholesky decomposition to provide a little more info when input is singular.\n\n## How was this patch tested?\n\nNew test case; jenkins tests.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #15177 from srowen/SPARK-11918.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/4ece1bf5be47f11c22da1d32181d72a2a4184f79"},{"sha":"b3e675a663c7c51a86131a8c2633ec713ab927c5","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-4563][CORE] Allow driver to advertise a different network address.\n\nThe goal of this feature is to allow the Spark driver to run in an\nisolated environment, such as a docker container, and be able to use\nthe host's port forwarding mechanism to be able to accept connections\nfrom the outside world.\n\nThe change is restricted to the driver: there is no support for achieving\nthe same thing on executors (or the YARN AM for that matter). Those still\nneed full access to the outside world so that, for example, connections\ncan be made to an executor's block manager.\n\nThe core of the change is simple: add a new configuration that tells what's\nthe address the driver should bind to, which can be different than the address\nit advertises to executors (spark.driver.host). Everything else is plumbing\nthe new configuration where it's needed.\n\nTo use the feature, the host starting the container needs to set up the\ndriver's port range to fall into a range that is being forwarded; this\nrequired the block manager port to need a special configuration just for\nthe driver, which falls back to the existing spark.blockManager.port when\nnot set. This way, users can modify the driver settings without affecting\nthe executors; it would theoretically be nice to also have different\nretry counts for driver and executors, but given that docker (at least)\nallows forwarding port ranges, we can probably live without that for now.\n\nBecause of the nature of the feature it's kinda hard to add unit tests;\nI just added a simple one to make sure the configuration works.\n\nThis was tested with a docker image running spark-shell with the following\ncommand:\n\n docker blah blah blah \\\n   -p 38000-38100:38000-38100 \\\n   [image] \\\n   spark-shell \\\n     --num-executors 3 \\\n     --conf spark.shuffle.service.enabled=false \\\n     --conf spark.dynamicAllocation.enabled=false \\\n     --conf spark.driver.host=[host's address] \\\n     --conf spark.driver.port=38000 \\\n     --conf spark.driver.blockManager.port=38020 \\\n     --conf spark.ui.port=38040\n\nRunning on YARN; verified the driver works, executors start up and listen\non ephemeral ports (instead of using the driver's config), and that caching\nand shuffling (without the shuffle service) works. Clicked through the UI\nto make sure all pages (including executor thread dumps) worked. Also tested\napps without docker, and ran unit tests.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #15120 from vanzin/SPARK-4563.","distinct":true,"url":"https://api.github.com/repos/zhengruifeng/spark/commits/b3e675a663c7c51a86131a8c2633ec713ab927c5"}]}
{"actor":{"display_login":"zhengruifeng"}}