{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14044","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14044/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14044/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14044/events","html_url":"https://github.com/apache/spark/pull/14044","id":163619305,"number":14044,"title":"[SPARK-16360][SQL] Speed up SQL query performance by removing redundant analysis in `Dataset`","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2016-07-04T07:36:50Z","updated_at":"2016-07-05T03:43:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14044","html_url":"https://github.com/apache/spark/pull/14044","diff_url":"https://github.com/apache/spark/pull/14044.diff","patch_url":"https://github.com/apache/spark/pull/14044.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nCurrently, there are a few reports about Spark 2.0 query performance regression for large queries.\r\n\r\nThis PR speeds up SQL query processing performance by removing redundant **consecutive** analysis in `Dataset.ofRows` function and `Dataset` instantiation. Specifically, this PR aims to reduce the overhead of SQL query analysis, not query execution. So, we can not see the result in the Spark Web UI. Please use the following query script. The result is **25.78 sec** -> **12.36 sec** as expected.\r\n\r\n**Sample Query**\r\n```scala\r\nval n = 4000\r\nval values = (1 to n).map(_.toString).mkString(\", \")\r\nval columns = (1 to n).map(\"column\" + _).mkString(\", \")\r\nval query =\r\n  s\"\"\"\r\n     |SELECT $columns\r\n     |FROM VALUES ($values) T($columns)\r\n     |WHERE 1=2 AND 1 IN ($columns)\r\n     |GROUP BY $columns\r\n     |ORDER BY $columns\r\n     |\"\"\".stripMargin\r\n\r\ndef time[R](block: => R): R = {\r\n  val t0 = System.nanoTime()\r\n  val result = block\r\n  println(\"Elapsed time: \" + ((System.nanoTime - t0) / 1e9) + \"s\")\r\n  result\r\n}\r\n```\r\n\r\n**Before**\r\n```scala\r\nscala> time(sql(query))\r\nElapsed time: 30.138142577s  // First query has a little overhead of initialization.\r\nres0: org.apache.spark.sql.DataFrame = [column1: int, column2: int ... 3998 more fields]\r\nscala> time(sql(query))\r\nElapsed time: 25.787751452s  // Let's compare this one.\r\nres1: org.apache.spark.sql.DataFrame = [column1: int, column2: int ... 3998 more fields]\r\n```\r\n\r\n**After**\r\n```scala\r\nscala> time(sql(query))\r\nElapsed time: 17.500279659s  // First query has a little overhead of initialization.\r\nres0: org.apache.spark.sql.DataFrame = [column1: int, column2: int ... 3998 more fields]\r\nscala> time(sql(query))\r\nElapsed time: 12.364812255s  // This shows the real difference. The speed up is about 2 times.\r\nres1: org.apache.spark.sql.DataFrame = [column1: int, column2: int ... 3998 more fields]\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nManual by the above script."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/230381891","html_url":"https://github.com/apache/spark/pull/14044#issuecomment-230381891","issue_url":"https://api.github.com/repos/apache/spark/issues/14044","id":230381891,"user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"created_at":"2016-07-05T03:43:55Z","updated_at":"2016-07-05T03:43:55Z","body":"Hi, @cloud-fan , @hvanhovell , @liancheng .\r\n\r\nAccording to @cloud-fan 's advice, after changing the following, it turns out that the difference is not noticeable.\r\n```\r\n-    new Dataset[Row](sparkSession, qe, RowEncoder(qe.analyzed.schema), skipAnalysis = true)\r\n+    new Dataset[Row](sparkSession, qe, RowEncoder(qe.analyzed.schema))\r\n```\r\n\r\nExactly as you guys told, the second call of `qe.assertAnalyzed()` is not the root cause. The only difference resides on `sparkSession.sessionState.executePlan(logicalPlan)`.\r\n\r\nI'll update the PR."}}
{"actor":{"display_login":"dongjoon-hyun"}}