{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15237","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15237/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15237/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15237/events","html_url":"https://github.com/apache/spark/pull/15237","id":179112808,"number":15237,"title":"[SPARK-17663] [CORE] SchedulableBuilder should handle invalid data access via scheduler.allocation.file","user":{"login":"erenavsarogullari","id":1437738,"avatar_url":"https://avatars.githubusercontent.com/u/1437738?v=3","gravatar_id":"","url":"https://api.github.com/users/erenavsarogullari","html_url":"https://github.com/erenavsarogullari","followers_url":"https://api.github.com/users/erenavsarogullari/followers","following_url":"https://api.github.com/users/erenavsarogullari/following{/other_user}","gists_url":"https://api.github.com/users/erenavsarogullari/gists{/gist_id}","starred_url":"https://api.github.com/users/erenavsarogullari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erenavsarogullari/subscriptions","organizations_url":"https://api.github.com/users/erenavsarogullari/orgs","repos_url":"https://api.github.com/users/erenavsarogullari/repos","events_url":"https://api.github.com/users/erenavsarogullari/events{/privacy}","received_events_url":"https://api.github.com/users/erenavsarogullari/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-09-25T22:21:26Z","updated_at":"2016-12-02T00:39:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15237","html_url":"https://github.com/apache/spark/pull/15237","diff_url":"https://github.com/apache/spark/pull/15237.diff","patch_url":"https://github.com/apache/spark/pull/15237.patch"},"body":"## What changes were proposed in this pull request?\n\nIf `spark.scheduler.allocation.file` has invalid `minShare` or/and `weight` values, these cause :\n- `NumberFormatException` due to `toInt` function\n- `SparkContext` can not be initialized.\n- It does not show meaningful error message to user.\n\nIn a nutshell, this functionality can be more robust by selecting one of the following flows :\n\n**1-** Currently, if `schedulingMode` has an invalid value, a warning message is logged and default value is set as `FIFO`. Same pattern can be used for `minShare`(default: 0) and `weight`(default: 1) as well\n**2-** Meaningful error message can be shown to the user for all invalid cases.\n\nPR offers :\n- `schedulingMode` handles just empty values. It also needs to be supported for **whitespace**, **non-uppercase**(fair, FaIr etc...) or `SchedulingMode.NONE` cases by setting default value(`FIFO`)\n- `minShare` and `weight` handle just empty values. They also need to be supported for **non-integer** cases by setting default values.\n- Some refactoring of `PoolSuite`.\n\n**Code to Reproduce :**\n\n```\nval conf = new SparkConf().setAppName(\"spark-fairscheduler\").setMaster(\"local\")\nconf.set(\"spark.scheduler.mode\", \"FAIR\")\nconf.set(\"spark.scheduler.allocation.file\", \"src/main/resources/fairscheduler-invalid-data.xml\")\nval sc = new SparkContext(conf)\n```\n\n**fairscheduler-invalid-data.xml :**\n\n```\n<allocations>\n    <pool name=\"production\">\n        <schedulingMode>FIFO</schedulingMode>\n        <weight>invalid_weight</weight>\n        <minShare>2</minShare>\n    </pool>\n</allocations>\n```\n\n**Stacktrace :**\n\n```\nException in thread \"main\" java.lang.NumberFormatException: For input string: \"invalid_weight\"\n    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n    at java.lang.Integer.parseInt(Integer.java:580)\n    at java.lang.Integer.parseInt(Integer.java:615)\n    at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)\n    at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)\n    at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$org$apache$spark$scheduler$FairSchedulableBuilder$$buildFairSchedulerPool$1.apply(SchedulableBuilder.scala:127)\n    at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$org$apache$spark$scheduler$FairSchedulableBuilder$$buildFairSchedulerPool$1.apply(SchedulableBuilder.scala:102)\n```\n## How was this patch tested?\n\nAdded Unit Test Case.\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/264340183","html_url":"https://github.com/apache/spark/pull/15237#issuecomment-264340183","issue_url":"https://api.github.com/repos/apache/spark/issues/15237","id":264340183,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T00:39:42Z","updated_at":"2016-12-02T00:39:42Z","body":"**[Test build #69506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69506/consoleFull)** for PR 15237 at commit [`4b6053f`](https://github.com/apache/spark/commit/4b6053f0146e44a3b9d4202dec2356be72059131).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}