{"action":"created","issue":{"url":"https://api.github.com/repos/SnappyDataInc/snappydata/issues/454","repository_url":"https://api.github.com/repos/SnappyDataInc/snappydata","labels_url":"https://api.github.com/repos/SnappyDataInc/snappydata/issues/454/labels{/name}","comments_url":"https://api.github.com/repos/SnappyDataInc/snappydata/issues/454/comments","events_url":"https://api.github.com/repos/SnappyDataInc/snappydata/issues/454/events","html_url":"https://github.com/SnappyDataInc/snappydata/pull/454","id":194100103,"number":454,"title":"[SNAP-1218] Enable RDD-bucket de-linking for single table and replicated table joins","user":{"login":"sumwale","id":11540628,"avatar_url":"https://avatars.githubusercontent.com/u/11540628?v=3","gravatar_id":"","url":"https://api.github.com/users/sumwale","html_url":"https://github.com/sumwale","followers_url":"https://api.github.com/users/sumwale/followers","following_url":"https://api.github.com/users/sumwale/following{/other_user}","gists_url":"https://api.github.com/users/sumwale/gists{/gist_id}","starred_url":"https://api.github.com/users/sumwale/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sumwale/subscriptions","organizations_url":"https://api.github.com/users/sumwale/orgs","repos_url":"https://api.github.com/users/sumwale/repos","events_url":"https://api.github.com/users/sumwale/events{/privacy}","received_events_url":"https://api.github.com/users/sumwale/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"hbhanawat","id":8359624,"avatar_url":"https://avatars.githubusercontent.com/u/8359624?v=3","gravatar_id":"","url":"https://api.github.com/users/hbhanawat","html_url":"https://github.com/hbhanawat","followers_url":"https://api.github.com/users/hbhanawat/followers","following_url":"https://api.github.com/users/hbhanawat/following{/other_user}","gists_url":"https://api.github.com/users/hbhanawat/gists{/gist_id}","starred_url":"https://api.github.com/users/hbhanawat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hbhanawat/subscriptions","organizations_url":"https://api.github.com/users/hbhanawat/orgs","repos_url":"https://api.github.com/users/hbhanawat/repos","events_url":"https://api.github.com/users/hbhanawat/events{/privacy}","received_events_url":"https://api.github.com/users/hbhanawat/received_events","type":"User","site_admin":false},"assignees":[{"login":"hbhanawat","id":8359624,"avatar_url":"https://avatars.githubusercontent.com/u/8359624?v=3","gravatar_id":"","url":"https://api.github.com/users/hbhanawat","html_url":"https://github.com/hbhanawat","followers_url":"https://api.github.com/users/hbhanawat/followers","following_url":"https://api.github.com/users/hbhanawat/following{/other_user}","gists_url":"https://api.github.com/users/hbhanawat/gists{/gist_id}","starred_url":"https://api.github.com/users/hbhanawat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hbhanawat/subscriptions","organizations_url":"https://api.github.com/users/hbhanawat/orgs","repos_url":"https://api.github.com/users/hbhanawat/repos","events_url":"https://api.github.com/users/hbhanawat/events{/privacy}","received_events_url":"https://api.github.com/users/hbhanawat/received_events","type":"User","site_admin":false},{"login":"soubhik-c","id":11516481,"avatar_url":"https://avatars.githubusercontent.com/u/11516481?v=3","gravatar_id":"","url":"https://api.github.com/users/soubhik-c","html_url":"https://github.com/soubhik-c","followers_url":"https://api.github.com/users/soubhik-c/followers","following_url":"https://api.github.com/users/soubhik-c/following{/other_user}","gists_url":"https://api.github.com/users/soubhik-c/gists{/gist_id}","starred_url":"https://api.github.com/users/soubhik-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soubhik-c/subscriptions","organizations_url":"https://api.github.com/users/soubhik-c/orgs","repos_url":"https://api.github.com/users/soubhik-c/repos","events_url":"https://api.github.com/users/soubhik-c/events{/privacy}","received_events_url":"https://api.github.com/users/soubhik-c/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-12-07T16:38:15Z","updated_at":"2016-12-08T06:57:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/SnappyDataInc/snappydata/pulls/454","html_url":"https://github.com/SnappyDataInc/snappydata/pull/454","diff_url":"https://github.com/SnappyDataInc/snappydata/pull/454.diff","patch_url":"https://github.com/SnappyDataInc/snappydata/pull/454.patch"},"body":"## Changes proposed in this pull request\r\n\r\n- added a LinkPartitionsToBuckets rule in optimizer phase to check for any non-replicated\r\n  table joins in the plan tree\r\n- use SnappySession contextObject to add the flag to set linkPartitionsToBuckets flag (thus disabling de-linking)\r\n- use the above flag for both the row/column RDD partitions\r\n  as well as plan's partitioning (OrderlessHashPartitioning)\r\n- updated NorthWind tests for above changes; also adding validation for expected partitions in joins\r\n  and a flag in NWQueries to just emit warning for partition expectation mismatch instead of failing\r\n\r\n## Patch testing\r\n\r\nprecheckin (in progress)\r\n\r\n## ReleaseNotes.txt changes\r\n\r\nNA\r\n\r\n## Other PRs \r\n\r\nNA\r\n"},"comment":{"url":"https://api.github.com/repos/SnappyDataInc/snappydata/issues/comments/265667514","html_url":"https://github.com/SnappyDataInc/snappydata/pull/454#issuecomment-265667514","issue_url":"https://api.github.com/repos/SnappyDataInc/snappydata/issues/454","id":265667514,"user":{"login":"sumwale","id":11540628,"avatar_url":"https://avatars.githubusercontent.com/u/11540628?v=3","gravatar_id":"","url":"https://api.github.com/users/sumwale","html_url":"https://github.com/sumwale","followers_url":"https://api.github.com/users/sumwale/followers","following_url":"https://api.github.com/users/sumwale/following{/other_user}","gists_url":"https://api.github.com/users/sumwale/gists{/gist_id}","starred_url":"https://api.github.com/users/sumwale/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sumwale/subscriptions","organizations_url":"https://api.github.com/users/sumwale/orgs","repos_url":"https://api.github.com/users/sumwale/repos","events_url":"https://api.github.com/users/sumwale/events{/privacy}","received_events_url":"https://api.github.com/users/sumwale/received_events","type":"User","site_admin":false},"created_at":"2016-12-08T06:57:09Z","updated_at":"2016-12-08T06:57:09Z","body":"@soubhik-c\r\n\r\n> actually IMO, SQLVal and Val should extend ConfigEntry\r\n\r\nNo they cannot since ConfigEntry is private[spark] and SQLConf itself is private[sql]. Hence the wrapper SQLConfigEntry with much cleaner functionality than provided by SQLConf.\r\n\r\n> Right now user code uses only the string variant\r\n> ideally it should be like\r\n\r\nNo, the latter one is private[sql] and not meant for users. In fact SQLConf itself is not visible so users cannot use it and have to embed the strings. For our case it is public so users can do:\r\n\r\n```scala\r\nsnContext.setConf(io.snappydata.Property.EnableExperimentalFeatures.name, \"true\")\r\n```\r\n\r\nCompare to current one in SQLConf which users can actually use:\r\n\r\n```scala\r\nsqlContext.setConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key, \"true\")\r\n```\r\n\r\nWhich is similar.\r\n\r\nSee the public type-safe usage:\r\n\r\n```scala\r\nProperty.HashJoinSize.get[Long](conf)\r\n```\r\n\r\nwhich is not available in SQLConf (all private).\r\n\r\nOr compare the declaration:\r\n\r\n```scala\r\n  val LocalHashJoinSize = SQLVal[Long](s\"${Constant.PROPERTY_PREFIX}localhashjoinsize\",\r\n    \"The join would be converted into a hash join if the table is of size less\" +\r\n        \"than localhashjoinsize. Default value is 100 MB.\", Some(100*1024*1024))\r\n```\r\n\r\nvs in SQLConf where explicit longConf/createWithDefault and unnecessary builder stuff is needed:\r\n\r\n```scala\r\n  val AUTO_BROADCASTJOIN_THRESHOLD = SQLConfigBuilder(\"spark.sql.autoBroadcastJoinThreshold\")\r\n    .doc(\"Configures the maximum size in bytes for a table that will be broadcast to all worker \" +\r\n      \"nodes when performing a join.  By setting this value to -1 broadcasting can be disabled. \" +\r\n      \"Note that currently statistics are only supported for Hive Metastore tables where the \" +\r\n      \"command<code>ANALYZE TABLE &lt;tableName&gt; COMPUTE STATISTICS noscan</code> has been run.\")\r\n    .longConf\r\n    .createWithDefault(10L * 1024 * 1024)\r\n```\r\n\r\nThis is an improvement in my opinion. If you have suggestions to better it, then let me know."}}
{"actor":{"display_login":"sumwale"}}