{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21495","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21495/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21495/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21495/events","html_url":"https://github.com/apache/spark/pull/21495","id":329315443,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkyNjAwNTM2","number":21495,"title":"[SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6","user":{"login":"dbtsai","id":1134574,"node_id":"MDQ6VXNlcjExMzQ1NzQ=","avatar_url":"https://avatars1.githubusercontent.com/u/1134574?v=4","gravatar_id":"","url":"https://api.github.com/users/dbtsai","html_url":"https://github.com/dbtsai","followers_url":"https://api.github.com/users/dbtsai/followers","following_url":"https://api.github.com/users/dbtsai/following{/other_user}","gists_url":"https://api.github.com/users/dbtsai/gists{/gist_id}","starred_url":"https://api.github.com/users/dbtsai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dbtsai/subscriptions","organizations_url":"https://api.github.com/users/dbtsai/orgs","repos_url":"https://api.github.com/users/dbtsai/repos","events_url":"https://api.github.com/users/dbtsai/events{/privacy}","received_events_url":"https://api.github.com/users/dbtsai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-06-05T06:17:32Z","updated_at":"2018-06-08T03:54:01Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21495","html_url":"https://github.com/apache/spark/pull/21495","diff_url":"https://github.com/apache/spark/pull/21495.diff","patch_url":"https://github.com/apache/spark/pull/21495.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nScala is upgraded to `2.11.12` and `2.12.6`. \r\n\r\nWe used `loadFIles()` in `ILoop` as a hook to initialize the Spark before REPL sees any files in Scala `2.11.8`. However, it was a hack, and it was not intended to be a public API, so it was removed in Scala `2.11.12`.\r\n\r\nFrom the discussion in Scala community, https://github.com/scala/bug/issues/10913 , we can use `initializeSynchronous` to initialize Spark instead. This PR implements the Spark initialization there.\r\n\r\nHowever, in Scala `2.11.12`'s `ILoop.scala`, in function `def startup()`, the first thing it calls is `printWelcome()`. As a result, Scala will call `printWelcome()` and `splash` before calling `initializeSynchronous`. \r\n\r\nThus, the Spark shell will allow users to type commends first, and then show the Spark UI URL. It's working, but it will change the Spark Shell interface as the following. \r\n\r\n```scala\r\n➜  apache-spark git:(scala-2.11.12) ✗ ./bin/spark-shell \r\nSetting default log level to \"WARN\".\r\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\r\nWelcome to\r\n      ____              __\r\n     / __/__  ___ _____/ /__\r\n    _\\ \\/ _ \\/ _ `/ __/  '_/\r\n   /___/ .__/\\_,_/_/ /_/\\_\\   version 2.4.0-SNAPSHOT\r\n      /_/\r\n         \r\nUsing Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)\r\nType in expressions to have them evaluated.\r\nType :help for more information.\r\n\r\nscala> Spark context Web UI available at http://192.168.1.169:4040\r\nSpark context available as 'sc' (master = local[*], app id = local-1528180279528).\r\nSpark session available as 'spark'.\r\n\r\n\r\nscala> \r\n```\r\n\r\nIt seems there is no easy way to inject the Spark initialization code in the proper place as Scala doesn't provide a hook. Maybe @som-snytt can comment on this.\r\n\r\nThanks.\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/395638832","html_url":"https://github.com/apache/spark/pull/21495#issuecomment-395638832","issue_url":"https://api.github.com/repos/apache/spark/issues/21495","id":395638832,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTYzODgzMg==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-06-08T03:54:01Z","updated_at":"2018-06-08T03:54:01Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91538/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}