{"action":"created","issue":{"url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1327","repository_url":"https://api.github.com/repos/TresAmigosSD/SMV","labels_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1327/labels{/name}","comments_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1327/comments","events_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1327/events","html_url":"https://github.com/TresAmigosSD/SMV/issues/1327","id":359248535,"node_id":"MDU6SXNzdWUzNTkyNDg1MzU=","number":1327,"title":"Extend smvJoinByKey api to handle null-safe join","user":{"login":"ninjapapa","id":8062822,"node_id":"MDQ6VXNlcjgwNjI4MjI=","avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ninjapapa","id":8062822,"node_id":"MDQ6VXNlcjgwNjI4MjI=","avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false},"assignees":[{"login":"ninjapapa","id":8062822,"node_id":"MDQ6VXNlcjgwNjI4MjI=","avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-09-11T22:32:55Z","updated_at":"2018-09-12T16:39:11Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Current `join` or `smvJoinByKey` will drop records with null value keys. See:\r\nhttps://stackoverflow.com/questions/41728762/including-null-values-in-an-apache-spark-join\r\n\r\nIt is the correct SQL behavior. However it will be convenience if we can have a null-safe join helper, so user don't need to worry about how to handle potential null-value keys. \r\n\r\n## Design\r\n\r\nExtend `smvJoinByKey` API with an optional parameter `isNullSafe` with default `false`. \r\n\r\nClient code\r\n```python\r\ndf1.smvJoinByKey(df2, ['k1', 'k2'], 'inner', true)\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/comments/420715965","html_url":"https://github.com/TresAmigosSD/SMV/issues/1327#issuecomment-420715965","issue_url":"https://api.github.com/repos/TresAmigosSD/SMV/issues/1327","id":420715965,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDcxNTk2NQ==","user":{"login":"ninjapapa","id":8062822,"node_id":"MDQ6VXNlcjgwNjI4MjI=","avatar_url":"https://avatars1.githubusercontent.com/u/8062822?v=4","gravatar_id":"","url":"https://api.github.com/users/ninjapapa","html_url":"https://github.com/ninjapapa","followers_url":"https://api.github.com/users/ninjapapa/followers","following_url":"https://api.github.com/users/ninjapapa/following{/other_user}","gists_url":"https://api.github.com/users/ninjapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ninjapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ninjapapa/subscriptions","organizations_url":"https://api.github.com/users/ninjapapa/orgs","repos_url":"https://api.github.com/users/ninjapapa/repos","events_url":"https://api.github.com/users/ninjapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ninjapapa/received_events","type":"User","site_admin":false},"created_at":"2018-09-12T16:39:11Z","updated_at":"2018-09-12T16:39:11Z","author_association":"CONTRIBUTOR","body":"@laneb Do you remember have we ever used the `broadcastOther` parameter in the client project?\r\nI know it is used internally by `smvSkewJoinByKey`.\r\n\r\nCurrent `smvJoinByKey` Scala interface is the following:\r\n```\r\n  def smvJoinByKey(\r\n      otherPlan: DataFrame,\r\n      keys: Seq[String],\r\n      joinType: String,\r\n      postfix: String = null,\r\n      dropRightKey: Boolean = true,\r\n      broadcastOther: Boolean = false\r\n  ): DataFrame\r\n```\r\n\r\nSince I python side only support the first 3 parameters, and I don't recall any client project is really using Scala interface. If so I will make current `smvJoinByKey` an private function, and make the Scala interface the same as the Python interface."}}
{"actor":{"display_login":"ninjapapa"}}