{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15792","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15792/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15792/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15792/events","html_url":"https://github.com/apache/spark/pull/15792","id":187614722,"number":15792,"title":"[SPARK-18295][SQL] Make to_json function null safe (matching it to from_json)","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-11-07T02:54:45Z","updated_at":"2016-11-08T00:54:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15792","html_url":"https://github.com/apache/spark/pull/15792","diff_url":"https://github.com/apache/spark/pull/15792.diff","patch_url":"https://github.com/apache/spark/pull/15792.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR proposes to match up the behaviour of `to_json` to `from_json` function for null-safety.\r\n\r\nCurrently, it throws `NullPointException` but this PR fixes this to produce `null` instead.\r\n\r\nwith the data below:\r\n\r\n```scala\r\nimport spark.implicits._\r\n\r\nval df = Seq(Some(Tuple1(Tuple1(1))), None).toDF(\"a\")\r\ndf.show()\r\n```\r\n\r\n```\r\n+----+\r\n|   a|\r\n+----+\r\n| [1]|\r\n|null|\r\n+----+\r\n```\r\n\r\nthe codes below\r\n\r\n```scala\r\nimport org.apache.spark.sql.functions._\r\n\r\ndf.select(to_json($\"a\")).show()\r\n```\r\n\r\nproduces..\r\n\r\n**Before**\r\n\r\nthrows `NullPointException` as below:\r\n\r\n```\r\njava.lang.NullPointerException\r\n  at org.apache.spark.sql.catalyst.json.JacksonGenerator.org$apache$spark$sql$catalyst$json$JacksonGenerator$$writeFields(JacksonGenerator.scala:138)\r\n  at org.apache.spark.sql.catalyst.json.JacksonGenerator$$anonfun$write$1.apply$mcV$sp(JacksonGenerator.scala:194)\r\n  at org.apache.spark.sql.catalyst.json.JacksonGenerator.org$apache$spark$sql$catalyst$json$JacksonGenerator$$writeObject(JacksonGenerator.scala:131)\r\n  at org.apache.spark.sql.catalyst.json.JacksonGenerator.write(JacksonGenerator.scala:193)\r\n  at org.apache.spark.sql.catalyst.expressions.StructToJson.eval(jsonExpressions.scala:544)\r\n  at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:142)\r\n  at org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:48)\r\n  at org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:30)\r\n  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\r\n```\r\n\r\n**After**\r\n\r\n```\r\n+---------------+\r\n|structtojson(a)|\r\n+---------------+\r\n|       {\"_1\":1}|\r\n|           null|\r\n+---------------+\r\n```\r\n\r\n\r\n## How was this patch tested?\r\n\r\nUnit test in `JsonExpressionsSuite.scala` and `JsonFunctionsSuite.scala`.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/259012480","html_url":"https://github.com/apache/spark/pull/15792#issuecomment-259012480","issue_url":"https://api.github.com/repos/apache/spark/issues/15792","id":259012480,"user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T00:54:11Z","updated_at":"2016-11-08T00:54:11Z","body":"Thanks, merging to master and 2.1"}}
{"actor":{"display_login":"marmbrus"}}